The Cowpath Chronicles 2: Is that a dog, a cat or a thunderstorm?
Blog: Capgemini CTO Blog
I live in a country that competes with the likes of Ireland and Seattle in getting the most rain in a year. The weather is a key topic to discuss at any occasion, and many people use their smartphones more frequently to check the latest forecasts than social media or email.
Clearly, forecasting the weather is a truly ancient skill. Experts would observe the flight of birds or the way smoke rose from a fire. Cows were believed to lie down to chew the cud when it was about to rain, in order to keep the ground underneath themselves dry. Cloud types would give weather indications: feathery cirrus clouds are a sign of an approaching warm front and coming rain, as is a sun gradually darkened by low cloud. Or, one would simply apply old wives’ tales – handed over from generation to generation – relying on best practices such as “red sky at night, shepherd’s delight. red sky in the morning, shepherd’s warning” or “rain before seven, fine by eleven.”
We have come a long way, with weather predictions nowadays based on solid science and up-to-date technologies. Still, even with a surge of sensors to collect weather data and ever more advanced prediction models, it turns out to be difficult to calculate the likelihood of precipitation or a local thunderstorm – even if it’s just for the forthcoming few hours or so. There are terabytes of data to ingest and lots of expensive, resource-consuming computing power needed to analyze it. Simulations and models are typically so complex and demanding that they can only be run a few times a day, limiting their real-time relevance. Also, the same computational demands limit the spatial resolution to about five kilometers, which is not sufficient for predicting weather patterns within urban areas and farmland.
Maybe, just maybe, we have spent a bit too much time on that long and winding cowpath.
AI researchers at Google certainly took a radically different road. They have applied machine learning models for highly accurate, localized precipitation forecasts that apply to the immediate future. This precipitation “nowcasting” focuses on zero- to six-hour forecasts with a one-kilometer resolution and a total latency of just five to ten minutes.
Their secret? A data-driven, “no physics harmed” approach, in which the machine learning models know nothing about atmospheric physics or any other metric pertaining to weather. Instead, it uses Convolutional Neural Networks (CNNs). That’s right, the same deep learning neural networks that are routinely used for image recognition and analysis, the same networks that can effectively tell a dog from a cat, estimate your age and emotional state of mind, or assess whether you are wearing a mask and keeping social distance from others.
Weather prediction is handled here as an image-to-image translation problem, simply learning from evolving patterns in large amounts of radar images. A significant advantage of this type of machine learning is that inference is fast and computationally cheap, given an already-trained model. The even more interesting news: turns out that Google’s nowcasting machine learning approach – even at its early stage of development – is already solidly beating even the most advanced, highly tuned, established weather forecast models. Admittedly this is not the case yet for long-term predictions, but that might only be a matter of time.
In a way, it is an intimidating thought – also to quite a few data scientists and weather scientists – to be able to accurately predict the weather without even remotely understanding the underlying physics, mechanisms, metrics, and attributes. The black-box nature of the next generation of AI systems provides raw power and tempting simplicity (as is also evident from a key trend in our TechnoVision series: How Deep Is Your Math), but to the price of limited explainability, and even less control – a theme that will regularly pop up as we further explore the perks and perils of AI-powered, hands-free, no-contact, self-driving, self-optimizing autonomous systems.
For now, Google’s nowcasting is yet again a great example – just like DeepMind’s AlphaGo was in the previous episode – of how innovative breakthroughs can be created by avoiding meandering cowpaths. Even if there are cows lying on it, chewing the cud.
Next up in the Cowpath Chronicles: why you no longer need to get together – even virtually – to run a successful innovation workshop. Stay tuned!