Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents

arXiv:2601.15322 · cs.AI, cs.CL · Submitted 2026-01-17 · Read on arXiv

cs.AI, cs.CL

Submitted: 2026-01-17

Updated: 2026-09-20

Comments: 23 pages, 4 figures, 8 tables. Corrects interpretation of historical results and clarifies study boundaries. Separate DFAH-Bench manuscript: arXiv:2607.20491. Code and data: https://github.com/ibm-client-engineering/output-drift-financial-llms. Original version accepted at the 2nd ICLR Workshop on Advances in Financial AI: Towards Agentic and Responsible Systems (ICLR 2026)

Code: https://github.com/ibm-client-engineering/output-drift-financial-llms

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 57× in this paper · explained in 1 episode
• Benchmark — 57× in this paper · explained in 2 episodes
• Alignment — 12× in this paper · explained in 1 episode
• Consistency — 10× in this paper · explained in 3 episodes
• Grounding — 7× in this paper · explained in 3 episodes
• Determinism-Faithfulness Assurance Harness — 6× in this paper
• Calibration — 5× in this paper · explained in 7 episodes
• Aggregation — 3× in this paper · explained in 1 episode
• Human-in-the-Loop — 3× in this paper · explained in 3 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Agentic Evaluation — 2× in this paper · explained in 1 episode
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Trace — 2× in this paper · explained in 2 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Agent Architectures — 1× in this paper · explained in 1 episode
• Anti-Money Laundering — 1× in this paper
• Core — 1× in this paper · explained in 2 episodes
• Data Provenance — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 3 episodes
• RAG — 1× in this paper · explained in 3 episodes
• Service Level Objective — 1× in this paper
• Verification — 1× in this paper · explained in 2 episodes

Sources

• GAIA: a benchmark for General AI Assistants
• BloombergGPT: A Large Language Model for Finance
• FinGPT: Open-Source Financial Large Language Models
• LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows
• Assessing Consistency and Reproducibility in the Outputs of Large Language Models: Evidence Across Diverse Finance and Accounting Tasks
• An Empirical Study of the Non-determinism of ChatGPT in Code Generation
• Large Language Models: An Applied Econometric Framework
• Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinations
• A Unified AI System For Data Quality Control and DataOps Management in Regulated Environments
• Evaluating Large Language Models in Scientific Discovery
• FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos
• Gorilla: Large Language Model Connected with Massive APIs

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection