AI Models Still Lag Behind Traditional Numerical Models in Predicting Sudden-Turning Typhoons

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "AI Models Still Lag Behind Traditional Numerical Models in Predicting Sudden-Turning Typhoons".

Jane: The paper was written by Daosheng Xu, Zebin Lu, Jeremy Cheuk-Hin Leung, Dingchi Zhao, Yi Li et al. from Guangzhou Institute of Tropical and Marine Meteorology and Guangdong Provincial Key Laboratory of Regional Numerical Weather Prediction and China Meteorological Administration and Key Laboratory of Physical Oceanography and Ministry of Education and Institute for Advanced Ocean Study and Frontiers Science Center for Deep Ocean Multispheres and Earth System and College of Oceanic and Atmospheric Sciences and Ocean University of China and College of Meteorology and Oceanography and National University of Defense Technology and Guangdong Meteorological Observatory and College of Atmospheric Science and Lanzhou University and National Meteorological Centre and State Key Laboratory of Tibetan Plateau Earth System and Resources and Environment and Institute of Tibetan Plateau Research and Chinese Academy of Sciences.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everyone. Today we're looking at a paper that's been making waves in the weather forecasting world, and the title really tells you everything you need to know: "AI Models Still Lag Behind Traditional Numerical Models in Predicting Sudden-Turning Typhoons."

Jane: Tom, that title is a bit of a gut punch, isn't it? I mean, we've been hearing so much about how AI weather models are catching up to, or even beating, the old-school physics-based models. And now this paper comes along and says, not so fast.

Tom: Exactly. And I love that they're being so specific. They're not saying AI is bad at everything. They're saying AI is worse at one very particular, very dangerous thing: typhoons that suddenly turn.

Jane: Right, and that's such an important distinction. A typhoon that just keeps going in a straight line is relatively easy to predict. But a typhoon that makes a sharp turn, like a car swerving without warning, that's the kind of thing that catches people off guard and causes real disasters.

Tom: And the paper points to a specific example, Typhoon Khanun in two thousand twenty-three. This thing made two sharp turns in five days as it passed through the Ryukyu Islands. And the AI model, Pangu-Weather, just couldn't keep up with the traditional European model.

Jane: So the title is basically saying, when it comes to the rare and the extreme, the old physics-based approach still has the edge. That's a really important reality check for anyone who thinks AI is just going to sweep in and solve everything.

Tom: It really is. And I think the key word in the title is "still." It's not saying AI will never get there. It's saying, right now, in this specific area, the traditional models are still the ones you want to trust.

Jane: And that's a good thing to know, because if you're a forecaster in a country that's about to get hit by a sudden-turning typhoon, you need to know which tool is going to give you the most reliable information.

Tom: Absolutely. So we've got the headline. But what's the actual evidence behind this claim? Let's dig into the details of the paper in the next segment.

Summary: Jane: So we're back, and we're still talking about "AI Models Still Lag Behind Traditional Numerical Models in Predicting Sudden-Turning Typhoons." Tom, let's get into what the researchers actually did.

Tom: Okay, so they looked at one hundred four typhoons in the Northwest Pacific from two thousand twenty to two thousand twenty-four. And they split them into three groups: ordinary ones, sudden-turning ones, and looping ones, which are the ones that do a little circle.

Jane: And that's a smart way to break it down, because it lets you see if the AI model's performance changes depending on how weird the typhoon's path is.

Tom: Right. And what they found is that for the ordinary typhoons, the AI model, Pangu-Weather, actually beat the traditional European model, ECMWF-IFS. The average error was about thirteen point five percent smaller. So AI is winning the easy cases.

Jane: And that's consistent with what the original Pangu-Weather paper claimed, right? That it's generally more accurate.

Tom: Exactly. But here's where it gets interesting. When they looked at the sudden-turning typhoons, the tables turned completely. The AI model's errors were about nine point four percent larger than the traditional model's. So on the hard cases, the physics-based model wins.

Jane: So it's not that AI is bad, it's that it's good at the common stuff and bad at the rare stuff. And the rare stuff is exactly what causes the most damage.

Tom: And they even looked at a specific case, Typhoon Khanun, which I mentioned earlier. The AI model's average track error over twenty-four to one hundred twenty hours was eighteen point eight percent worse than the traditional model. And the gap just got bigger as the forecast lead time increased.

Jane: So the longer the forecast, the worse the AI model does relative to the traditional one. That's a big deal, because you need those long lead times to evacuate people.

Tom: And here's the kicker. They also compared the AI model to human forecasters. And for the first couple of days, the AI model was better than the humans. But by day five, the difference basically disappeared. So the AI model's advantage fades away exactly when you need it most.

Jane: That's a really sobering finding. So the summary is, AI is great at the average, but it's not ready for the extreme. And the paper has some ideas about why that is, which we should get into.

Tom: Yeah, let's talk about that in the next segment. What's causing this blind spot?

Improvements: Tom: We're back with "AI Models Still Lag Behind Traditional Numerical Models in Predicting Sudden-Turning Typhoons." And Jane, we've established that AI is struggling with the sudden turns. Now let's talk about why, and what the paper suggests we do about it.

Jane: Right. So the researchers found a really interesting pattern. The typhoons that the AI model predicted well were the ones that were basically being pushed along by a strong, steady wind, the western Pacific subtropical high. It's like a conveyor belt.

Tom: And when that conveyor belt is strong, the typhoon just follows it. The AI model is great at predicting the big picture, the large-scale weather patterns. So it nails those cases.

Jane: But when the conveyor belt is weak, the typhoon's path depends much more on its own internal structure. The fine details of the storm itself. And that's where the AI model falls apart.

Tom: And the paper argues that's because AI models are trained to match historical data, but they don't actually simulate the physics. They don't have a real understanding of the storm's core, the updrafts, the downdrafts, the way the pressure and wind interact.

Jane: And they found evidence of that. The AI model actually failed to predict the strong vertical winds in the typhoon. It just smoothed them out. And it also had trouble with the relationship between wind and pressure, which is a fundamental physical law.

Tom: So it's like the AI model is a brilliant artist who can paint a beautiful landscape, but if you ask it to paint a single leaf in a storm, it just smears it. It doesn't understand how the leaf actually moves.

Jane: That's a great analogy. So what do they suggest? How do we fix this?

Tom: They suggest two main things. First, we need higher-resolution training data. If the AI model has never seen the fine details of a typhoon's core, it can't learn to predict them. So we need better reanalysis datasets.

Jane: And the second thing?

Tom: The second thing is to add physical constraints to the AI model. Instead of just letting it learn from data, we need to build in the laws of physics. So the model is forced to produce results that are physically possible.

Jane: So it's not just about throwing more data at it. It's about teaching it the rules of the game.

Tom: Exactly. And that's a really important insight, because it suggests that the future of weather forecasting isn't either AI or physics. It's a combination of both.

Jane: And that's a much more hopeful message than "AI is bad." It's saying AI has a specific weakness, and we know how to fix it.

Tom: Right. So let's wrap this up in our final segment.

Conclusion: Tom: So we've spent this whole episode on "AI Models Still Lag Behind Traditional Numerical Models in Predicting Sudden-Turning Typhoons." And I think the big takeaway is that AI weather models are incredibly powerful, but they have a specific blind spot.

Jane: And that blind spot is the rare, extreme events. The sudden turns, the unusual paths. The things that don't happen very often, so the AI model hasn't seen enough examples to learn them.

Tom: And the paper shows that for those cases, the traditional physics-based models are still the ones you want to trust. Especially when you need a forecast more than two or three days out.

Jane: But it's not a defeat. It's a roadmap. The paper gives us clear directions on how to improve AI models, with better data and physical constraints.

Tom: And that's the exciting part. This isn't the end of the story. It's a challenge. A challenge to the AI community to build models that don't just handle the average, but also the extreme.

Jane: And that's so important, because as the climate changes, we're likely to see more of these weird, unpredictable weather events. We need tools that can handle them.

Tom: So we're saying goodbye to this paper, but we're taking its message with us. AI is the future, but it's not the present. Not yet.

Jane: And that's a good thing to remember. We need to keep investing in the traditional models while we push the AI models to get better.

Tom: Alright, that's it for this one. Thanks for listening, everyone. We'll be back soon with another paper to dig into. See you then.

Jane: Bye everyone!