FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

arXiv:2609.25192 · cs.CL · Submitted 2026-09-21 · Read on arXiv

cs.CL

Submitted: 2026-09-21

Updated: 2026-09-21

Comments: 20 pages, 3 figures, and 7 tables. Dataset available at https://huggingface.co/datasets/inclusionAI/FinFIRST

Code: https://github.com/Tencent-Hunyuan/Hy3

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 48× in this paper · explained in 2 episodes
• Verification — 19× in this paper · explained in 2 episodes
• Large Language Model — 11× in this paper
• Benchmarking — 8× in this paper · explained in 3 episodes
• Alignment — 6× in this paper · explained in 1 episode
• Consistency — 6× in this paper · explained in 3 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Agentic Search — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Model-Agnostic — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Sparse Attention — 1× in this paper · explained in 5 episodes

Sources

• Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks
• BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent
• GLM-5: from Vibe Coding to Agentic Engineering
• Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
• FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning
• FinanceBench: A New Benchmark for Financial Question Answering
• Kimi K3: Open Frontier Intelligence
• FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks
• From Scores to Skills: A Cognitive Diagnosis Framework for Evaluating Financial Large Language Models
• MiniMax Sparse Attention
• CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model
• BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs
• FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models
• GAIA: a benchmark for General AI Assistants
• MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application
• Humanity's Last Exam
• Measuring short-form factuality in large language models
• BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
• FinBen: A Holistic Financial Benchmark for Large Language Models
• FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering