A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Models

Graph neural networks (GNNs) are routinely employed for short-range forecasting on multivariate time series with a spatial graph structure, but this work critically audits widely adopted benchmark datasets and evaluation protocols to uncover structural biases that may lead to misleading performance comparisons. The gist is that analyzing these benchmarks via classical time series methods reveals why spatially-unaware linear models pose a stronger competitor than previously reported, suggesting over-reliance on these datasets for method comparison is flawed.

Statistical Analysis of Benchmark Datasets

The study performs a statistical analysis of lagged temporal and spatial correlations on five benchmark datasets: Chickenpox, PedalMe, WikiMaths, METR-LA, and PEMS-BAY. This analysis involves computing three types of correlation: temporal correlation (autocorrelation), spatial correlation (linear relationship between node i at time t and its neighbors j at time t−τ), and partial spatial correlation, which estimates the predictive power of lagged spatial information conditional on the state of neighboring nodes. The findings reveal that for datasets like Chickenpox, the median partial spatial correlation is very close to zero at all lags, indicating almost no spatial information, while traffic datasets like METR-LA and PEMS-BAY show a strong, slowly decaying process with a long memory in both the temporal and spatial correlations.

Critique of Evaluation Protocols

The research scrutinizes existing evaluation protocols to identify shortcomings. A key finding is that many studies compare models using differenced rather than raw time series, which yields inferior predictive performance across the board. Specifically, for Chickenpox, undoing the differencing reveals that the temporal correlation function changes from being non-existent to a persistent function with correlations lasting until at least lag 10, and its spatial signal gains much more linear informativeness. This suggests that models trained on undifferenced data can generalize better, as evidenced by a smaller ratio of test MSE to train MSE on the raw Chickenpox dataset.

Re-evaluation of Baselines

The paper re-evaluates simple baselines, motivating the use of spatially uninformed models such as SARIMA (seasonal ARIMA) on Chickenpox and WikiMaths. The analysis shows that for these datasets, SARIMA can outperform many larger models. For instance, on WikiMaths, SARIMA outperforms existing state-of-the-art models. Furthermore, the authors propose a novel approach: training of GNNs on SARIMA residuals as a training target that can raise predictive performance to the level of state-of-the-art methods on METR-LA and PEMS-BAY.

Performance Comparison with Linear Models

The work investigates simple linear baselines, including Persistence, Historical average, AR(H) per node, RidgeVAR, and DLinear. The results show that among the best baselines for Chickenpox and PedalMe are AR(H) per node and DLinear, despite being spatially unaware. For WikiMaths, RidgeVAR is coming first and even placing above the state-of-the-art TDE-GNN. The synthetic experiment further tests this by showing that the performance gap between GRUGCN and ARIMA is positively influenced by the variance of temporal coefficients across all nodes in the graph, while spatial coefficient variance was not found to be significant.

Conclusion and Recommendations

The study concludes that benchmark datasets are strongly driven by the temporal signals. The authors recommend reducing over-reliance on these datasets for method comparison and advocating for more rigorous statistical evaluation. They suggest that PedalMe and Chickenpox should remain undifferenced when used for benchmarking purposes, as this preserves the low-frequency signal. Finally, they demonstrate how combining SARIMA with a spatial GNN model can reach the level of state-of-the-art performance on traffic datasets like METR-LA and PEMS-BAY. The work also hypothesizes that linear models are effective because they do not implicitly assume that the data they are modelling have a homogenous spatiotemporal response to the past signal, unlike GNNs.

Outlook

The authors suggest future work should focus on developing GPU-optimized backends for node-wise fitting, such as AutoARIMA or Cuml. Additionally, the paper points toward integrating SARIMA with GNNs as a promising avenue for developing novel forecasting methods. The synthetic experiment provides a framework for understanding how heterogeneity in nodal time series properties affects the performance of both linear models and GNNs. The authors also highlight that "the global spatial coefficient also seems to be the only parameter that has a positive effect on GNN performance, perhaps because it is global, and hence homogeneous.

Here are the specific improvements to AI systems derived from this research, categorized by the technical intervention:

)1. Enhanced Pre-processing for Benchmark Datasets (Chickenpox & PedalMe):

By recommending against default differencing for Chickenpox and PedalMe datasets, the improved system should:

• Perform forecasting models directly on the raw, undifferenced time series data rather than automatically applying first-order differencing (as done by default in libraries like PyTorch Geometric Temporal).

• Utilize a more rigorous evaluation protocol that compares model performance on both raw and differenced versions to detect overdifferencing artifacts.

)2. Implementation of Residual Learning for GNNs:

The system should incorporate a two-stage modeling pipeline for spatiotemporal forecasting:

• Perform initial forecasting using a SARIMA model (which is effective at capturing temporal correlation).

• Calculate the residuals from this SARIMA prediction.

• Train a Graph Neural Network (GNN) on these SARIMA residuals, augmented with past observed errors and the predicted values from the SARIMA step.

)3. Adaptation to Heterogeneous Temporal Dynamics:

The improved GNN architecture should be explicitly designed to handle node-specific temporal memory variations:

• Instead of assuming uniform temporal coefficients across all nodes, the model should leverage mechanisms (like those explored in Section 4.2 and Table 6) that allow for the variance of temporal coefficients across nodes to influence performance. This could involve parameterizing the AR component or using attention mechanisms sensitive to node-specific historical dependencies.

)4. Improved Baseline Selection Strategy:

The system's baseline selection process should be refined based on statistical correlation analysis:

• Prioritize baselines that are statistically informed by temporal and spatial correlations (as quantified in Section 2.1), rather than relying solely on simple, spatially uninformed models like historical averages or basic RidgeVAR, especially when the data shows strong residual spatial information (like METR-LA/PEMS-BAY).

• Implement a dynamic selection mechanism that suggests SARIMA as a strong linear baseline for smaller datasets (Chickenpox/WikiMaths) and potentially SARIMA with fixed orders for larger traffic datasets to ensure fairness across different data scales.

)5. Leveraging Synthetic Experiments for Model Robustness:

The system development lifecycle should incorporate synthetic experiments based on the proposed discrete random graph process (Equation 8):

• Use these simulations to test the robustness of GNNs against node-level temporal heterogeneity (varying node temporal coefficients, spatial weights, and global connectivity).

• This ensures that when deploying GNNs on real data exhibiting complex spatial dependencies, they are better equipped to handle the inherent variability in how different nodes evolve over time.

Graph neural networks (GNNs) are routinely employed for spatiotemporal forecasting, yet their performance across widely used benchmark datasets is inconsistent. Here, we perform an audit of dataset properties and baseline models to assess the quality of the benchmarks, and the robustness of the conclusions drawn from them. Using classical statistical tools, we characterise spatiotemporal lagged dependencies in benchmarks, and examine how temporal differencing changes these relationships and affects model rankings. Motivated by this, we re-evaluate temporal linear baselines, significantly reducing the apparent gains from GNNs on several benchmarks, and surpassing GNNs on others. Suspecting that GNNs struggle to extract linear, node-wise signals, we find that supplying them with autoregressive residuals improves their performance particularly on non-traffic benchmarks. Finally, controlled synthetic experiments reveal that GNNs are sensitive to heterogeneity in temporal dynamics and spatial graph interactions. Together, our findings demonstrate that baseline specification, data pre-processing and system heterogeneity shape the interpretations drawn from benchmark rankings, informing the design and robust evaluation of GNNs.