The Cowpath Chronicles 2 – Is that a dog, a cat, or a thunderstorm?
Blog: Capgemini CTO Blog
I live in a country that competes with the likes of Ireland and Seattle in getting most rain in a year. The weather is a key topic to discuss at any occasion, and many people use their smartphones more frequently to check the latest forecasts than social media or e-mail.
Clearly, forecasting the weather is a truly ancient skill. Experts would look at the flight of the birds or the way smoke rose from a fire. Cows were believed to lie down to chew the cud when it was about to rain, in order to keep the ground underneath themselves dry. Cloud types would give weather indications: feathery cirrus clouds are a sign of an approaching warm front and coming rain, as is a sun gradually darkened by low cloud. Or one would simply apply old wives’ tales – handed over from generation to generation – relying on best practices such as “Red Sky At Night, Shepherd’s Delight. Red Sky In The Morning, Shepherd’s Warning” or “Rain Before Seven, Fine By Eleven”.
We have come a long way, with weather predictions nowadays based on solid science and up-to-date technologies. Still, even with a surge of sensors to collect weather data and ever more advanced prediction models, it turns out difficult to calculate the likelihood of precipitation or a local thunderstorm – even if it’s just for the forthcoming few hours or so. There are terabytes of data to ingest and lots of expensive, resource-consuming computing power needed to analyze it. Simulations and models are typically so complex and demanding that they can only be run a few times a day, limiting their real-time relevance. Also, the same computational demands limit the spatial resolution to about 5 kilometers, which is not sufficient for predicting weather patterns within urban areas and farmland.
Maybe, just maybe, we have spent a bit too much time on that long and winding cowpath.
AI researchers at Google certain took a radically different road. They have applied machine learning models for highly accurate, localized precipitation forecasts that apply to the immediate future. This precipitation ‘nowcasting’ focuses on 0 to 6 hour forecasts with a 1 km resolution and a total latency of just 5 to 10 minutes.
Their secret? A data-driven, ‘no physics harmed’ approach, in which the machine learning models knows nothing about atmospheric physics or any other metric pertaining to weather. Instead it uses Convolutional Neural Networks (CNNs). That’s right, the same deep leaning neural networks that are routinely used for image recognition and analysis, the same networks that can effectively tell a dog from a cat, can estimate your age and emotional state of mind, or can assess whether you ae wearing a mouth mask and keeping your distance from others.
Weather prediction is handled here as an image-to-image translation problem, simply learning from evolving patterns in large amounts of radar images. A significant advantage of this type of machine learning is that inference is fast and computationally cheap, given an already-trained model. The even more interesting news: turns out that Google’s nowcasting machine learning approach – even at its early stage of development – is already solidly beating even the most advanced, highly tuned established weather forecast models. Admittedly this is not the case yet for long-term predictions, but that might be a matter of time.
In a way, it is an intimidating thought – also to quite a few data scientists and weather scientists – to be able to accurately predict the weather without even remotely understanding the underlying physics, mechanisms, metrics and attributes. The Black Box nature of the next generation of AI systems provides raw power and tempting simplicity (as is also evident from a key trend in our TechnoVision series: ‘How Deep Is Your Math‘), but to the price of limited explainability, and even less control. A theme that regularly will pop up, as we further explore the perks and perils of AI-powered, handsfree, no-contact, self-driving, self-optimizing autonomous systems.
For now, Google’s nowcasting is yet again a great example – just like DeepMind’s AlphaGo was in the previous episode – of how innovative breakthroughs can be created by avoiding meandering cowpaths. Even if there are cows lying on it, chewing the cud.
Next up in the Cowpath Chronicles: why you no longer need to get together – even virtually – to run a successful innovation workshop. Stay tuned!