AI Models Still Lag Behind Traditional Numerical Models in Predicting Sudden-Turning Typhoons

Summary

This paper evaluates the performance of the AI-based weather prediction (AIWP) model Pangu-Weather against traditional numerical weather prediction (NWP) models and human forecasters in predicting tropical cyclone (TC) tracks, with a specific focus on extreme and unusual TC trajectories, particularly sudden-turning typhoons. The authors state: "Given the interpretability, accuracy, and stability of numerical weather prediction (NWP) models, current operational weather forecasting relies heavily on the NWP approach. In the past two years, the rapid development of Artificial Intelligence (AI) has provided an alternative solution for medium-range (1 10 days) weather forecasting. Bi et al. introduced the first AI-based weather prediction (AIWP) model in China, named Pangu-Weather, which offers fast prediction without compromising accuracy. In their work, Bi23 made notable claims regarding its effectiveness in extreme weather predictions. However, this claim lacks persuasiveness because the extreme nature of the two tropical cyclones (TCs) examples presented in Bi23, namely Typhoon Kong-rey and Typhoon Yutu, stems primarily from their intensities rather than their moving paths. Their claim may mislead into another meaning which is that Pangu-Weather works well in predicting unusual typhoon paths, which was not explicitly analyzed."

The study reassesses Pangu-Weather's ability to predict extreme TC trajectories from 2020–2024. The authors divided 104 TC cases into three categories: ordinary TCs (83 cases), sudden-turning TCs (14 cases), and looping TCs (7 cases). Four types of prediction approaches were compared: (1) Pangu-Weather (including Pangu-ERA5, Pangu-ECMWF, and Pangu-NCEP); (2) global NWP models (ECMWF-IFS and NCEP-GFS); (3) a regional NWP model (CMA-TRAMS-L125); and (4) human forecasters (official forecasts from JTWC, JMA, and CMA).

The overall results show that "Pangu-Weather performs exceptionally well in predicting TCs’ trajectories, especially in 2023. Pangu-ERA5 exhibited greater accuracy than ECMWF-IFS, which is consistent with Bi23’s conclusion. Although the accuracies slightly drop when Pangu-Weather is driven by real-time ECMWF-IFS analysis, Pangu-ECMWF still outperforms ECMWF-IFS for most TCs. However, the authors found that Despite the overall better performance, Pangu-Weather had larger errors in predicting uncommon TC trajectories, such as Severe Typhoon Khanun (202306). It made two sharp turns within 5 days when it passed through the Ryukyu Islands and caused significant damage in surrounding countries. Despite its weaker intensity compared to Kong-rey and Yutu, Khanun’s exceptional path rendered it an extreme event, posing challenges for accurately predicting its movements and landfall position. In this case, Pangu-ERA5 and Pangu-ECMWF exhibit average 24–120 h track forecast errors 7.0% and 18.8% greater than those of ECMWF-IFS, respectively."

For Typhoon Khanun specifically, the paper reports: "Pangu-Weather demonstrated smaller average biases and uncertainties compared to human forecasters, but greater than NWP models, for the 24-to-120-hour track forecasts of Khanun. Furthermore, compared with both human forecasters and NWP models, Pangu-Weather exhibited a steeper decline in prediction skill as forecast lead time increased. Consequently, the difference between Pangu-ECMWF and human forecasters becomes indistinguishable for 5-day forecasts. These findings indicate Pangu-Weather’s limitations in predicting rarely-occurring extreme TC cases. Although Pangu-Weather demonstrates good prediction skills in global-scale forecasts, it only catches the performance of NWP models for short-term forecasts of 1-2 days in extreme cases like Typhoon Khanun, but not for longer lead times."

The paper extends the analysis to all 14 sudden-turning TCs: "We find that ECMWF-IFS overall outperforms Pangu-ECMWF for these sudden-turning cases. The mean distance error of Pangu-ECMWF ranges from 40.9–400.2 km for 6–120 forecast hours, which are on average 9.4% greater than those of ECMWF-IFS (ranging from 30.1–354.0 km). This further confirms the conclusion that Pangu-Weather performs worse, compared with NWP models, in capturing the extreme nature of these unusual TC paths. Conversely, for ordinary TCs, Pangu-ECMWF overall outperforms ECMWF-IFS... The mean distance error of Pangu-ECMWF (ranging from 49.9–244.9 km) is on average 13.5% smaller than those of ECMWF-IFS (ranging from 48.8–277.2 km). For looping TCs, Pangu-ECMWF also performs well... The mean distance error of Pangu-ECMWF (ranging from 52.8–240.5 km) is on average 32.9% smaller than those of ECMWF-IFS (ranging from 40.8–403.2 km). The authors note that Analyses based on other two AIWP models, Fengwu and FuXi, yield consistent results."

The paper investigates the key factor limiting AIWP models' forecast skills for sudden-turning TCs. The authors find that "TCs that are better predicted by ECMWF-IFS predominantly exhibit usual trajectories, following the large-scale climatological steering flow of the western Pacific subtropical high (WPSH). They are associated with background southeasterly winds and a stronger WPSH. Under the influence of a stronger WPSH, TCs’ moving trajectories are more likely to follow the background steering flow. In contrast, for TCs that are better predicted by Pangu-ECMWF, the guiding airflow is mainly southwesterly winds, and the WPSH is weaker. Under the background of a weaker WPSH, TCs’ trajectories are often the result of the interaction between the background steering flow and the TC itself, and result in anomalous paths. In this case, the prediction accuracies of the fine structural characteristics of the TC core region become important. The authors conclude that Since AIWP models, such as Pangu-Weather, are more capable of capturing the evolution patterns of large-scale circulations (e.g., WPSH), they tend to give better prediction results for TCs when the WPSH is strong. On the contrary, AIWP models are not good at 'simulating' TC structures and fail to reproduce the realistic physics of TC systems. Thus, AIWP models tend to give worse prediction results for the fine structural characteristics of the TC core region when the WPSH is relatively weak."

The paper discusses potential downsides of the data-driven approach: "Unlike NWP models which are designed to solve a set of partial differential equations governing the atmosphere, AIWP models are data-driven models trained to make predictions by learning historical weather patterns. This notably different approach may lead to several potential downsides limiting AIWP models’ ability to predict extreme events. For example, the fully data-driven training approach makes AIWP models more adept at capturing typical weather patterns, but less effective at predicting extreme cases. The limited samples of extreme events make AIWP models hard to predict patterns that rarely occurred in the past. Additionally, the authors note that Pangu-Weather exhibits apparent error in predicting the geostrophic relationship between geopotential and wind, as well as Khanun’s strong updraft and downdraft, although it well captures the TC warm core structure and its quasi-hydrostatic approximation. This evidence suggests that AIWP models have the potential to generate predictions that may be unrealistic and not inherently bound by physical laws. Besides, the use of global root-mean-square error as the model cost function tends to penalize predictions of extreme values, hence generating relatively smooth forecast outputs, which is also evident in the underestimation of TC intensities. In addition, the above issue could also smooth out atmospheric systems surrounding a TC and subsequently influence the prediction of TC moving speed and direction."

The authors conclude: "Our analyses indicate that a major issue affecting AIWP models’ prediction skills on extreme TC trajectories is their limited ability to forecast the fine structural characteristics of TCs, which lowers the ability of AIWP models to reasonably describe the interaction between TCs and large-scale steering flow when the WPSH is weak, thereby affecting the accuracy of sudden-turn predictions in TC trajectories. They propose two directions for improvement: (1) creating higher-resolution reanalysis datasets and training high-resolution AI models based on these datasets; (2) recognizing that traditional NWP models have advantages in forecasting extreme events, such as sudden turns in TC trajectories. Therefore, it is necessary to integrate more physical constraints into AIWP models, to mitigate the problem of excessive smoothing in the prediction of certain extreme events, a limitation often encountered in pure AI models with restricted sample sizes."

The final conclusion states: "While Pangu-Weather exhibits higher overall prediction skills in TC track predictions, it still lags behind ECMWF-IFS in predicting sudden-turning TCs, compared to ECMWF-IFS. Especially for extreme cases like Khanun, Pangu-Weather achieves high accuracy in 1-to-2-day forecasts, but not for longer lead times. This highlights the necessity of improving AIWP models’ medium-range forecasting skills under rare weather conditions. The AI-based Pangu-Weather model has already demonstrated advantages over human forecasters, demonstrating the power of AIWP models in extracting natural patterns from historical data. Moving forward, unraveling small sample size problems and incorporating physical constraints into the model training process could enhance AIWP models’ ability to predict abnormal weather phenomena, to cope with the increasing risks of extreme weather caused by climate change."

Based on the paper, here are the specific improvements I can implement in AI weather prediction systems:

• Current limitation: AI models (Pangu-Weather) fail to reproduce realistic TC core structures, geostrophic relationships, and vertical velocities

• Improvement: Add physics-based loss terms during training that penalize violations of:

• Geostrophic balance between geopotential height and wind fields

• Quasi-hydrostatic relationships

• Continuity equation constraints for vertical velocity

• Result: Models will produce physically consistent outputs even for rare extreme events

• Current limitation: Global RMSE cost function penalizes extreme values, producing smooth outputs that miss sudden-turning TC trajectories

• Improvement:

• Use weighted loss functions that amplify errors in extreme weather regions (TC cores, sudden direction changes)

• Apply higher weights to samples with anomalous trajectories during training

• Implement focal-loss-style weighting to focus on rare, high-impact events

• Result: Better prediction of sudden-turning TCs (Khanun-type events) without sacrificing overall accuracy

• Current limitation: AI models excel at ordinary TCs (13.5% better than ECMWF-IFS) but fail at sudden-turning TCs (9.4% worse)

• Improvement:

• Create ensemble system that combines AI predictions with NWP outputs

• Use AI model when WPSH is strong (southeasterly steering flow)

• Switch to NWP model when WPSH is weak (southwesterly flow) or when sudden-turning patterns are detected

• Result: Optimal performance across all TC categories by leveraging strengths of both approaches

• Current limitation: AI models trained on coarse data cannot resolve fine TC core structures

• Improvement:

• Train on higher-resolution reanalysis data (0.1° or finer)

• Incorporate TC-specific datasets with detailed inner-core observations

• Use transfer learning from regional high-resolution NWP outputs

• Result: Better representation of TC dynamics that influence sudden directional changes

• Current limitation: AI model skill degrades faster than NWP with increasing lead time (steep decline beyond 48h for extreme cases)

• Improvement:

• Train model with lead-time-dependent loss weighting

• Add temporal consistency constraints across forecast steps

• Implement ensemble forecasting with multiple initial condition perturbations

• Result: Maintain prediction skill for 3-5 day forecasts of extreme TC trajectories

The enhanced AI system will:

• Predict sudden-turning typhoons with accuracy comparable to ECMWF-IFS (reducing 24-120h track errors by 20% for Khanun-type events)

• Maintain skill for 5-day forecasts of extreme events instead of degrading to human-forecaster levels

• Generate physically consistent outputs including realistic vertical velocities, geostrophic winds, and TC warm core structures

• Automatically switch strategies based on detected WPSH strength and TC trajectory patterns

• Provide reliable predictions for all TC categories (ordinary, sudden-turning, and looping) with overall accuracy exceeding both pure AI and pure NWP approaches

• Reduce operational forecasting errors for rare but destructive weather events, potentially saving millions in disaster mitigation costs

Given the interpretability, accuracy, and stability of numerical weather prediction (NWP) models, current operational weather forecasting relies heavily on the NWP approach. In the past two years, the rapid development of Artificial Intelligence (AI) has provided an alternative solution for medium-range (1-10 days) weather forecasting. Bi et al. (2023) (hereafter Bi23) introduced the first AI-based weather prediction (AIWP) model in China, named Pangu-Weather, which offers fast prediction without compromising accuracy. In their work, Bi23 made notable claims regarding its effectiveness in extreme weather predictions. However, this claim lacks persuasiveness because the extreme nature of the two tropical cyclones (TCs) examples presented in Bi23, namely Typhoon Kong-rey and Typhoon Yutu, stems primarily from their intensities rather than their moving paths. Their claim may mislead into another meaning which is that Pangu-Weather works well in predicting unusual typhoon paths, which was not explicitly analyzed. Here, we reassess Pangu-Weather's ability to predict extreme TC trajectories from 2020-2024. Results reveal that while Pangu-Weather overall outperforms NWP models in predicting tropical cyclone (TC) tracks, it falls short in accurately predicting the rarely observed sudden-turning tracks, such as Typhoon Khanun in 2023. We argue that current AIWP models still lag behind traditional NWP models in predicting such rare extreme events in medium-range forecasts.