FinVerse: Financial Time-Series Benchmark

arXiv:2608.03259 · cs.LG, cs.AI · Submitted 2026-08-20 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "FinVerse: Financial Time-Series Benchmark Toward A More Realistic Evaluation for Financial Time-series Forecasting".

Jane: The paper was written by Jaehoon Lee, Jun Seo, Seunghan Lee, Tae Yoon Lim, Dongwan Kang et al. from LG AI Research.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Tom: Now, let's summarize what the paper tells us about the state of AI forecasting using FinVerse. The core finding seems to be that strong performance on generic benchmarks doesn't guarantee useful results in financial forecasting.

Jane: It’s a sobering finding, Tom, because it suggests that simply training a large foundation model isn't enough if you don't test it against real-world decision criteria.

Lu: The paper shows this by demonstrating that the correlation between generic error-based benchmarks and FinVerse is only moderate—a Pearson r of zero point four zero for the matched models.

Meng: That forty percent correlation is significant; it means a lot of what we expect from a big AI model based on old tests simply doesn't translate to real utility in practice.

Lalam: This realization highlights that the current benchmark landscape, while useful, isn' FinVerse gives us the necessary complexity to see where generic performance falls short when applied to financial forecasting.

Tom: And Jane mentions that by organizing these one hundred sixteen thousand eight hundred ninety-seven series into a hierarchy—inter-country, country-level, and individual—they are capturing the full spectrum of what AI models need to handle.

Jane: It’s not just a dump of data; it' structured according to its economic meaning. For example, crypto assets require different handling than fixed income indicators.

Lu: The sixty thousand two hundred thirty-two target series were manually selected because their semantics make them vital for financial decision-making, which is a huge qualitative step in the evaluation design.

Meng: From an operational standpoint, this means we can’t treat all time series as interchangeable; the model must learn to respond differently depending on whether it' forecasting a regional index or a specific company's earnings.

Lalam: The impact here is that we are moving toward an era where AI models are tested for their practical competence rather than just for their statistical accuracy, which is a major step in elevating the standard of AI capability.

Tom: It sounds like the summary is that generic metrics just don't capture the nuance needed to predict how a portfolio or a market will behave.

Jane: Definitely. We're going to look at the specific ways FinVerse achieves this "more realistic evaluation" next, so stick around for Segment three.

Improvements: Tom: So, how does FinVerse achieve this more realistic evaluation? The authors suggest three key things that improve the testing process.

Jane: They aren't just using one metric; they are looking at point-wise forecast quality, cross-sectional ranking quality, and realized portfolio performance as distinct evaluation aspects.

Lu: I find the concept of cross-sectional ranking particularly fascinating—evaluating how a model ranks multiple assets simultaneously is a much richer test than just seeing if it predicts one price accurately.

Meng: The portfolio backtesting aspect is what catches my eye, because instead of just averaging errors, we're simulating a real-world strategy where the AI picks the top ten percent fraction based on its predictions.

Lalam: This simulates risk and return trade-offs in a way that generic metrics simply cannot, showing us how AI can directly contribute to better financial decision making.

Tom: And Jane points out that for many specific types of series, like CPI or stock direction, simple error minimization is useless; you might care more about the direction or even the acceleration of change.

Jane: That’s right. The point-wise metrics are tailored—we have hit ratio (HR) and its variants to capture directional correctness, which is often what matters most for price movements.

Lu: For macroeconomic indicators, we can't just look at if it goes up or down; we need to know if the rate of change is accelerating or decelerating, which the and metrics handle beautifully.

Meng: And then, instead of just focusing on prediction accuracy for each single asset, the IC metric allows us to see how well-ranked a whole portfolio would be based on those cross-sectional predictions.

Lalam: The implications are that FinVerse is not just a test; it's a blueprint for building AI that understands financial causality and practical utility, which is crucial for cultural advancement in decision making.

Tom: It’s clear the authors are advocating for domain-aware benchmarks, moving beyond the one-size-fits-all approach.

Jane: We've seen how FinVerse addresses this by assigning specific metrics to specific series based on their economic role, setting up a really sophisticated evaluation environment for our listeners.

Conclusion: Tom: So, let’s wrap up the discussion on "FinVerse: Financial Time-Series Benchmark Toward A More Realistic Evaluation for Financial Time-series Forecasting" and what it means for the future of AI.

Jane: The key takeaway is that we have a tool that allows us to judge AI models based on their real-world utility, not just their mathematical score, which is a huge win.

Lu: The results show that performance isn't just about model size; smaller or medium-sized models can perform very well across certain evaluation aspects, suggesting the quality of the pretraining data matters immensely.

Meng: From an engineering standpoint, it seems like building a robust system now requires not just massive scale but also designing it to handle these diverse metrics and economic contexts.

Lalam: This benchmark is a foundational piece that ensures AI development aligns with practical human needs, promoting a culture of responsible and useful technological progress across all sectors.

Tom: The findings clearly show that the three evaluation aspects—point-wise accuracy, cross-sectional ranking, and portfolio backtesting—are complementary rather than redundant.

Jane: They are distinct ways to prove a model's value, which is something we need to keep in mind as we look toward future releases of this benchmark.

Lu: The authors are also planning to extend this framework into other domains like medicine and energy, showing that this is just the first step in a broader Time-Series Benchmark Universe.

Meng: I see the practical impact of having clear direction for future work; if we know exactly what "good" looks like across different financial series, we can design better training loops.

Lalam: The structure of FinVerse allows us to see AI not just as a predictor, but as a functional tool that supports reliable decision making in a global economy.

Tom: And Jane sums it up by saying the power of FinVerse is that no single model dominates all criteria, reinforcing the idea that these different financial objectives are genuinely distinct and require careful consideration.

Jane: It’s been a fascinating deep dive into this paper, Tom. We'll have to see what other researchers come up with when we look at the next big thing in AI.

Conclusion: Tom: So, we’ve been through FinVerse, but what's the absolute core message here? Essentially, this paper shows that simply having high scores on generic time-series benchmarks doesn't mean an AI model will actually be useful for making smart financial decisions.

Jane: It really highlights that the old methods of evaluation are too broad; they don't capture the nuance needed to predict how a portfolio or a market will behave in real life.

Lu: And from a research perspective, it’s fascinating because the correlation between these generic tests and FinVerse is only moderate—a Pearson r of zero point four zero for the matched models. That tells us that ninety-six percent of what we expect from previous tests simply doesn' not transfer to practical utility.

Meng: That's a huge operational insight; it means an engineer can’t just rely on old metrics and move toward a better production system, because the model quality changes dramatically depending on whether we are testing direction or portfolio performance.

Lalam: The impact is that we are moving toward an era where AI development must be tested for its practical competence, ensuring that the technological progress aligns with reliable human decision-making in our global economy.

Tom: Exactly, so FinVerse gives us a powerful tool to judge AI models based on their real-world utility, not just their mathematical score.

Jane: We’ve seen how it structures the financial data—by organizing one hundred sixteen thousand eight hundred ninety-seven series into categories like inter-country or individual—to make sure the evaluation is tailored to the economic meaning of each asset.

Lu: I’m particularly excited about how they are extending this to other domains, such as medicine and energy, showing that this framework is just a universal standard for benchmarking AI.

Meng: For me, it's a massive practical step toward designing systems that can support stable outcomes across different financial scales without being overwhelmed by sheer scale.

Lalam: This is about elevating the standard of AI capability; ensuring that our tools don't just provide answers, but provide *useful* answers in a global system.

Tom: It sounds like we are officially moving beyond the one-size-fits-all approach to a more sophisticated, domain-aware evaluation.

Jane: We've seen how FinVerse addresses this by assigning specific metrics—like hit ratio or Sharpe ratio—to specific series based on their economic role, setting up a really sophisticated testing environment for our listeners.

Lu: This is the future; it’s not just about accuracy anymore, but about proving functional value.

Meng: And it' clear that the authors are making us think about how we actually *use* the AI in a practical investment strategy rather than just how well it predicts a single point.

Lalam: It's been a fascinating look at this shift; realizing that no single model dominates all criteria, reinforcing the need for careful consideration across distinct financial objectives.

Tom: So, while FinVerse offers us this complete framework for understanding AI's practical utility in finance, we still have some technical limitations to address.

Jane: That’s right; we need to make sure our next discussion tackles those implementation details.

Jaehoon Lee, Jun Seo, Seunghan Lee, Tae Yoon Lim, Dongwan Kang, Hwanil Choi, Minjae Kim, Sungdong Yoo, Junhyeok Kang, Sangjun Han, Soonyoung Lee, Wonbin Ahn

LG AI Research

cs.LG, cs.AI

Submitted: 2026-08-20

Updated: 2026-08-21

Comments: 24 pages

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 87/100

The gist: FinVerse: Financial Time-Series Benchmark Toward A More Realistic Evaluation for Financial Time-series Forecasting The emergence of time-series foundation models necessitates robust evaluation

Key concepts

FinVerse: FinVerse is a comprehensive benchmark designed to evaluate AI models for financial time-series forecasting. It organizes data into a hierarchy (e.g., inter-country or individual assets) and uses tailored evaluation metrics, moving beyond a one-size-fits-all approach to test the practical utility of AI models.
Generic Benchmarks: The hosts discuss the limitations of traditional AI testing methods. The core finding is that generic error-based benchmarks only show a moderate correlation (Pearson r of 0.40) with real-world financial performance, meaning much of what we expect from large AI models does not translate to practical utility.
Evaluation Aspects: FinVerse uses three distinct ways to test model quality: point-wise forecast quality, cross-sectional ranking quality, and realized portfolio performance. This allows evaluators to see how well a model ranks multiple assets simultaneously or how it performs in a simulated real-world investment strategy.

Terminology

Benchmark — 86× in this paper · explained in 2 episodes
Foundation Model — 24× in this paper · explained in 3 episodes
Accuracy — 22× in this paper · explained in 1 episode
Foundation Models — 17× in this paper · explained in 7 episodes
Time Series Forecasting — 9× in this paper · explained in 3 episodes
Transformer — 9× in this paper · explained in 2 episodes
alpha — 8× in this paper · explained in 1 episode
Aggregation — 6× in this paper · explained in 1 episode
Alignment — 3× in this paper · explained in 1 episode
Information Coefficient — 3× in this paper
Core — 2× in this paper · explained in 2 episodes
Distribution Shift — 2× in this paper · explained in 6 episodes
Fine-Tuning — 2× in this paper · explained in 5 episodes
Adaptation — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 4 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Generalization — 1× in this paper · explained in 13 episodes
In-Context Learning — 1× in this paper · explained in 4 episodes
Meta-learning — 1× in this paper · explained in 2 episodes
Representation Learning — 1× in this paper · explained in 2 episodes
Robustness — 1× in this paper · explained in 5 episodes

Summary

FinVerse: Financial Time-Series Benchmark Toward A More Realistic Evaluation for Financial Time-series Forecasting

The emergence of time-series foundation models necessitates robust evaluation protocols that can distinguish between models merely achieving generic statistical accuracy from models that are genuinely useful for real-world decision making. Existing time-series forecasting benchmarks, such as GIFT-Eval, provide standardized comparisons but evaluate heterogeneous series with uniform error-based metrics. This limitation raises a practical question: if a decision maker selects the best-performing model on such a benchmark, should they expect reasonable performance on the decisions that actually depend on the forecast? The authors argue that the meaning of a good forecast changes across domains and target series, concluding that strong performance under such metrics does not necessarily imply that a model’s forecasts will support the best real-world decisions across domains.

To address this gap, FinVerse is introduced as a domain-specific benchmark initiative toward more realistic evaluation. FinVerse is designed around the principle that each time series should be evaluated according to the decision context implied by its economic meaning, rather than treating financial forecasting as a homogeneous error minimization problem.

Scale and Structure of FinVerse

The released data artifact contains 116,897 financial time series with 171.1M observations. A fixed evaluation set consisting of 60,232 economically meaningful target series (with 17.4M observations) is selected based on their relevance to financial forecasting and decision making. The remaining historical data prior to the evaluation start date (2020-01-01) are used to construct a training set of 66,134 time series and 100.3M observations for foundation model pretraining and fine-tuning.

FinVerse organizes financial time series across a broad hierarchy of scopes and categories:

Scope: Inter-country (e.g., Foreign Exchange, Crypto), Country (e Fixed Income, Macro Indicators), and Individual (e Equity, Fundamentals).
The benchmark covers diverse financial time series including equity prices, exchange rates, commodities, crypto assets, fixed-income indicators, market indicators.

** FinVerse Evaluation Methodology**

FinVerse moves beyond uniform error minimization by defining 11 metric families comprising 78 evaluation metrics. It assigns the most appropriate evaluation metrics to each individual time series based on its underlying economic meaning. This approach evaluates models across three complementary aspects:

Point-wise Forecast Quality: This includes directional correctness (Hit Ratio, HR, and squared HR) and adapted scaled error metrics (MASEr el, MASEabs, and MASEor i). These variants capture both the direction of movement and the accuracy of predicting raw values.
Cross-sectional Ranking Quality: This is measured using the Information Coefficient (IC), which assesses the Spearman rank correlation between predicted and realized future changes across assets in groups where relative ordering is meaningful (e.g, currencies or equities).
Realized Portfolio Performance: The benchmark converts forecasts into a long-only strategy, rebalancing at defined intervals. Metrics calculated for this aspect include annualized return, annualized volatility, Sharpe ratio, and maximum drawdown (MDD).

** Empirical Findings**

The authors evaluated 43 public time-series forecasting foundation models using FinVerse. Key findings include:

Our analysis of 43 public time-series forecasting foundation models shows that strong performance under generic forecasting criteria does not necessarily translate into useful financial forecasts. This highlights the need for domain-aware benchmarks.
The performance across the three evaluation aspects is complementary rather than redundant. Point-wise and cross-sectional rankings are strongly related (r = 0.72), but the point-wise–portfolio relationship is weaker (r = 0.37).
No single model consistently dominates all metric families; model quality varies substantially depending on the financial objective being measured. For instance, Reverso Small achieves the best overall FinVerse rank, followed by Chronos-2 Synth, TiRex 1.1 and TimesFM 2.5 200M.

The authors conclude that FinVerse provides a practical and informative view of model performance in financial forecasting by replacing one-size-fits-all error-based evaluation with decision-oriented metrics tailored to the underlying economic meaning of each financial time series.

Improvements for AI systems

Based on the rigorous analysis of the FinVerse benchmark—a paradigm shift from generic error minimization to decision-oriented evaluation—the following improvements can be implemented in AI systems designed for financial time-series forecasting.

The system will move beyond simple Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE). The training objectives must incorporate the directional and magnitude characteristics defined by FinVerse:

Directional Loss Integration (HR, HR, squared HR): The loss function will be augmented to penalize not just the magnitude of error, but also the failure to predict the direction of change. This is crucial for asset prices.
System Capability: The model will prioritize predicting whether a price will rise or fall (directional correctness) over predicting the exact price level, making it highly effective in markets where directional movement is more profitable than precise timing.
Relative Change Loss (MASEr el): The system will incorporate a loss function that penalizes errors relative to a no-change baseline, ensuring the model is optimized for predicting returns rather than raw values.
System Capability: The model will be better suited for high-frequency or volatile assets where small percentage changes (returns) are the primary driver of value.
Persistence Baseline Error (MASEor i): A loss component will penalize deviation from the most recent observed value, ensuring the model understands inertia and short-term stability in a market.
System Capability: The model will demonstrate increased robustness in stable market conditions, avoiding overreactions to minor fluctuations.

The system's evaluation pipeline will incorporate and optimize for the Information Coefficient (IC) derived from FinVerse:

Relative Ordering Optimization: The training objective will include a component that optimizes the Spearman rank correlation between predicted and realized changes across a set of correlated assets (e.g., various commodities or crypto pairs).
System Capability: The model will be able to reliably identify which assets are trending relative to others, enabling it to outperform simple single-asset predictors in portfolio construction.

The system will be evaluated using the comprehensive metrics derived from the backtesting simulation:

Risk-Adjusted Return Optimization (Sharpe Ratio): The training pipeline will incorporate a reward function that rewards high cumulative net asset value relative to its associated volatility.
System Capability: The model will generate forecasts that not only predict high returns but also achieve these returns with controlled, sustainable risk levels, ensuring the predictions are practical for real-world investment strategies.
Drawdown Minimization (MDD): The system will be penalized for large sequential losses during simulated rebalancing cycles.
System Capability: The model will learn to avoid high-risk trading strategies that lead to catastrophic capital loss, making it suitable for risk-averse institutional deployment.

The evaluation framework itself is improved by adopting FinVerse's rank-based aggregation strategy:

Heterogeneous Metric Ranking: Instead of averaging raw metric scores (which would allow large categories to dominate), the system's performance will be evaluated using the geometric mean of ranks across three distinct pillars (Point-wise, IC, Portfolio).
System Capability: The model will demonstrate consistent usefulness across diverse financial domains (e.g., comparing its performance on a global index versus its performance on individual equities), reflecting genuine utility rather than just success in the most data-dense categories.

Summary of Improvements: The system will transition from being a predictor to an investment advisor, optimizing not just for the accuracy of its prediction, but for the utility and risk profile of its advice.

Abstract

As time-series foundation models have emerged, the need for benchmarks that can evaluate their forecasting ability in meaningful ways has become increasingly important. Existing time-series forecasting benchmarks provide useful standardized comparisons, but they often evaluate heterogeneous series with uniform error-based metrics. Strong performance under such metrics does not necessarily imply that a model's forecasts will support the best real-world decisions across domains. For example, in stock forecasting, correctly predicting whether a price will rise or fall can be more directly relevant to realized returns than minimizing point-wise forecast error alone. To this end, we introduce FinVerse, a finance-domain time-series forecasting benchmark that takes a first step toward more realistic evaluation. The released FinVerse data artifact contains 116,897 financial time series with 171.1M observations, of which 60,232 series with 17.4M observations are selected as evaluated targets based on their economic relevance to financial decisions. Unlike generic forecasting benchmarks that primarily emphasize uniform point-forecast or probabilistic accuracy, FinVerse defines 11 metric families comprising 78 evaluation metrics and assigns the most appropriate evaluation metrics to each individual time series based on its underlying economic meaning. Our analysis of 43 public time-series forecasting foundation models shows that strong performance under generic forecasting criteria does not necessarily translate into useful financial forecasts. This finding highlights the need for domain-aware benchmarks that evaluate models under objectives closer to real-world decision making.

Sources

Holistic Evaluation of Language Models