FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings

arXiv:2608.07400 · cs.AI, cs.DB, econ.GN, q-fin.EC, q-fin.GN · Submitted 2026-08-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings".

Jane: The paper was written by Sasan Mansouri, Daniel Saad, Mark Wahrenburg, Manu Weissel and Fabian Woebbeking from University of Groningen and Goethe University Frankfurt and DataNXT GmbH and Halle Institute for Economic Research and Martin Luther University Halle-Wittenberg.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back, everyone. Today's paper is called "FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings," and it comes from Sasan Mansouri at the University of Groningen, Daniel Saad, Mark Wahrenburg, and Manu Weissel at Goethe University Frankfurt, and Fabian Woebbeking at the Halle Institute for Economic Research and Martin Luther University Halle-Wittenberg. It's a benchmark for financial QA over SEC filings, and the emphasis on evidence grounding in the title is really the whole story.

Jane: I'm glad you read out the full author list, because DataNXT, a Frankfurt-based company, is on it too. That mix of university research and a firm that builds financial eye tools shows up all over the design. This feels built for analyst workflows rather than for a leaderboard.

Tom: The core problem is that in SEC filings a plausible answer can be grounded in the wrong evidence. The authors argue the primary bottleneck in automated financial analysis is evidence discrimination rather than answer composition — telling apart the target firm's disclosure from a competitor's near-identical boilerplate, or from a prior period's filing. In practice, that wrong evidence is a passage that looks correct at first glance.

Jane: That rings true. Risk-factor sections in 10-Ks are heavily templated, so two pharmaceutical companies can describe their litigation exposure in almost the same words. If a system can't tie the answer to the correct filing, the answer is worthless even when it's numerically correct.

Lu: The paper spans three sectors — pharmaceuticals, oil and gas, and automotive — across 22 companies and filings from 2024 and 2025. Those confusable passages aren't hypothetical. They're real competitor filings sitting right next to the correct evidence in the same corpus.

Meng: And the authors flip the usual priority. General-domain QA treats supporting evidence as a secondary annotation, but here an answer's validity depends on strict provenance to the underlying disclosure. For an auditor or a compliance reviewer, that distinction is what separates useful output from a liability. The paper's whole design principle is that you can't evaluate financial answers without knowing where they come from.

Lalam: That's the broader significance, from where I sit. Regulated finance cares about provenance as much as correctness, and once you put eye into audit or compliance, a citation pointing at the wrong company can undermine the whole output and the trust placed in it, even if the answer reads perfectly. That's why this evidence-first framing matters beyond any single benchmark.

Tom: So the team built a benchmark directly from that pain point. There are 1185 manually written questions, and each one carries gold evidence plus a set of deliberately confusing distractors. Let's look at what is actually inside the dataset.

Summary: Tom: We've set the stage — the authors built this dataset around the evidence problem. Now, what's actually in it? There are 1185 question–answer records, all manually authored over the 10-K and 10-Q filings of 22 companies. Five business students did the writing: they read the filings, wrote every question, wrote every reference answer, and transcribed the supporting passages by hand, with page references.

Jane: And they didn't use a language model for any of that, which is a meaningful design choice

Paper discussion segment 3: Tom: So to recap, FinRank is a benchmark that checks whether financial QA systems can find the right evidence in SEC filings, not just the right answers.

Jane: And that focus on evidence points to a few concrete improvements the authors want the field to adopt. The biggest one is simple: stop grading financial answers without checking where the evidence came from.

Tom: Right, because a citation pointing to the wrong company's filing is worse than no citation at all in audit or compliance work. The paper basically argues that provenance should be a first-class metric.

Jane: They also suggest practical fixes, not just evaluation ones. The metadata filter is a good example — when the system knows which ticker, year, and form it should be looking at, BM25 jumps from 32 to 55 percent recall at ten.

Tom: That's huge, and it tells you that deployed tools should use filing metadata aggressively before doing any semantic search. It's cheap, deterministic, and it eliminates most of the confusable distractors upfront.

Jane: And for the harder cases that survive filtering, the paper recommends stratified reporting. Aggregate scores hide the fact that multi-passage and 10-Q questions are much harder across the board.

Tom: That's an important implication for anyone building on this — if you only look at your mean Recall@10, you'll miss that your model collapses on quarterly filings or multi-source synthesis.

Jane: The implications go beyond model development, too. In regulated settings, an answer with the right citation is actionable; an answer with the wrong citation is a liability. FinRank makes that trade-off measurable.

Tom: Which could push vendors of financial eye tools to actually show evidence-attribution numbers in their marketing, instead of just answer accuracy.

Jane: Exactly. And the authors leave the door open for more: they haven't run generative RAG baselines yet, and they explicitly call for a double-annotation study to verify label correctness.

Tom: So the natural next step for the benchmark is human adjudication at scale. That's the kind of work that would make FinRank even harder to dismiss in high-stakes deployment.

Jane: And we'll get into what that future work might look like, and whether this benchmark can hold up outside the three sectors it covers, right after the break.

Paper discussion segment 4: Tom: So far we've covered the dataset's design and the changes the authors want the field to adopt — now let's look at how they open the paper, because the abstract does some clever framing before you even get to the numbers.

Jane: It does. The opening sentence sets up a contrast: most financial QA benchmarks grade answer correctness, but FinRank grades whether a correct-looking answer is grounded in the right evidence. That immediately shifts the goal posts.

Tom: They call that "evidence discrimination" — the primary bottleneck in automated financial analysis. Not answer composition, but telling apart the target firm's disclosure from a competitor's near-identical boilerplate. That idea runs through the whole abstract.

Jane: And they argue this is what makes financial QA fundamentally different from open-domain QA. In general search, there's usually one obvious source for a fact. In SEC filings, the same standard accounting language appears across companies and across reporting periods.

Tom: Right, that's why they also flag document length and structural complexity. A single 10-K spans hundreds of pages, mixing accounting, legal, and forward-looking language — and answers often depend on evidence scattered across tables, footnotes, and narrative sections. That complicates retrieval in a way open-domain benchmarks don't capture.

Jane: The abstract also telegraphs the headline results. Even a 7B instruction-tuned embedder reaches only 44 point 8 percent recall at ten on the pooled evidence set. That's a strong statement about how hard this benchmark actually is.

Tom: And the smaller models barely beat BM25 — at most 3 point 5 points of Recall@10 — while a finance-adapted embedder trails BM25 by almost ten points. The authors are making the point that domain labels on embeddings don't automatically transfer to templated filing text.

Jane: That result should make anyone building finance-specific encoders sit up. It suggests the real gains come from scale or from adapting to SEC filings specifically, not just any financial corpus.

Tom: Exactly. And the abstract ends with the hard-negative finding: pairwise accuracy drops 13 to 20 points when you replace random negatives with the curated ones. That's the empirical proof that these distractors are genuinely confusable.

Jane: Which brings us to how those hard negatives were actually constructed and why the "same industry, different company" bucket dominates — we'll dig into that next.

Conclusion: Tom: So today we've been talking about FinRank, a benchmark that grades financial QA systems on whether they find the right evidence in SEC filings rather than just the right-looking answer.

Jane: And honestly, the most striking part is how hard even strong systems find that task. The best model in the paper, a 7B instruction-tuned embedder, only reaches about 45 percent recall at ten on the pooled evidence set.

Tom: That number should surprise people who think retrieval is basically solved. And it gets worse — a finance-adapted embedding model actually trails plain BM25 by nearly ten points. So throwing domain labels at the problem doesn't automatically help on templated filing text.

Jane: The authors show that knowing the target filing's metadata — ticker, year, form type — is far more powerful than any semantic trick. Filtering to the right filing lifts BM25 from 32 to 55 percent recall at ten, which is a huge jump for zero model changes.

Tom: But even with that filter, about half the gold evidence is still missed. That tells you the remaining challenge is genuine semantic discrimination between near-identical disclosures, not just corpus size.

Jane: And that's where the hard negatives come in. Every model drops 13 to 20 points of pairwise accuracy when you swap random distractors for human-curated ones. Those numbers validate the whole dataset design.

Tom: The authors are also upfront about limits. The benchmark is small, skewed toward 10-Ks and qualitative questions, and single-annotator without formal agreement statistics. That's why they push stratified reporting so hard.

Jane: Their roadmap is clear: run actual RAG generation baselines, do a double-annotation audit, and broaden coverage beyond three sectors. All of that would make FinRank even more credible for high-stakes use.

Tom: And for now, the practical message for anyone building financial assistants is simple — check your evidence, not just your answer. FinRank gives you the tool to do exactly that.

Jane: Great note to end on. Next up, we're switching gears to a paper that tries to bypass document retrieval altogether by feeding models curated vendor data directly through the Model Context Protocol. We'll see whether that shortcut holds up.

Sasan Mansouri, Daniel Saad, Mark Wahrenburg, Manu Weissel, Fabian Woebbeking

University of Groningen · Goethe University Frankfurt · DataNXT GmbH · Halle Institute for Economic Research · Martin Luther University Halle-Wittenberg

cs.AI, cs.DB, econ.GN, q-fin.EC, q-fin.GN

Submitted: 2026-08-07

Updated: 2026-08-10

Comments: 24 pages, 3 figures. Dataset and evaluation code: https://github.com/datanxt/FinRank

Code: https://github.com/datanxt/FinRank

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 65/100

The gist: The paper introduces FinRank, a benchmark for financial question answering and retrieval over SEC 10-K and 10-Q filings.

Key concepts

Evidence discrimination
The ability to tell apart the correct evidence from similar but wrong passages, such as a competitor's boilerplate or a prior year's filing. The paper argues this is the main bottleneck in financial QA, not generating the answer itself.
Hard negatives
Distractor passages that are deliberately similar to the correct evidence, like a competitor's risk-factor section. Replacing random negatives with these curated ones drops model pairwise accuracy by 13–20 points, showing they are genuinely confusable.
Metadata filtering
Using known filing metadata—ticker, year, form type—to narrow the search space before semantic retrieval. This simple step lifts BM25 recall@10 from 32% to 55%, showing that deterministic filtering is more effective than complex embeddings for this task.
Provenance
The source or origin of an answer, specifically the exact passage in an SEC filing. In regulated finance, an answer with the wrong citation is a liability, so FinRank treats provenance as a first-class metric rather than a secondary annotation.

Terminology

Summary

The paper introduces FinRank, a benchmark for financial question answering and retrieval over SEC 10-K and 10-Q filings. The authors argue that financial QA evaluation focuses too heavily on answer correctness, whereas in SEC filings, a plausible and even numerically correct answer can be grounded in the wrong evidence. They note that "regulatory disclosures are heavily templated. Standardized accounting conventions, risk-factor disclosures, and revenue-recognition notes read almost identically across competing firms and reporting periods. Consequently, the primary bottleneck in automated financial analysis is rarely answer composition, but evidence discrimination—distinguishing the target firm's disclosure from plausible distractors, such as a competitor's near-identical boilerplate or a prior period's filing."

The authors position FinRank against existing benchmarks: "General-domain QA benchmarks (e.g., Rajpurkar et al., 2016; Kwiatkowski et al., 2019), by contrast, treat supporting evidence as a secondary annotation—inverting the requirements of financial domain QA, where an answer's validity depends on strict provenance to its underlying disclosure. They also state that existing financial benchmarks (Z. Chen, W. Chen, et al., 2021; Zhu et al., 2021; Reddy et al., 2024) focus primarily on numerical calculation over provided snippets or end-to-end correctness, failing to isolate evidence retrieval and hard-negative suppression."

FinRank consists of 1,185 question–answer records manually authored over the 10-K and 10-Q filings of 22 companies across three sectors (pharmaceuticals, oil and gas, and automotive), covering filing years 2024–2025. The sector breakdown is: pharmaceuticals (499 records), oil and gas (438), and automotive (248), with per-company counts ranging from 9 (XOM) to 142 (JNJ).

Each record pairs a question and reference answer with:

  • Gold supporting passages (with page numbers)

  • Question- and filing-level metadata (topic, difficulty, reasoning type, evidence scope)

  • A hand-curated set of hard-negative passages drawn from confusable passages within filings, across reporting periods, and across comparable firms

  • Gold sub-question decompositions (query rewrites) for 69% of records

The topic distribution is deliberately near-uniform across eight topics (Company Overview, Financials, Footnotes, Governance, Accounting, Legal, Risk, and Shareholder Return), with 141–158 records each. Difficulty follows a 30%/40%/30% Easy/Medium/Hard split. Document type is skewed: 88% 10-K and 12% 10-Q; filing year is 75% 2025 and 25% 2024; reasoning type is 68% qualitative vs. 32% quantitative; evidence scope spans single- (34%), two- (40%), and multi-passage (26%) records.

The authors claim four contributions:

  1. First financial QA benchmark with curated hard negatives: "to our knowledge, the first financial QA benchmark to release a dedicated set of human-selected, semantically confusable hard negatives for each question, enabling direct comparison of discrimination against curated versus random distractors."

  2. Rich, stratifiable annotations: "across four axes (document type, reasoning type, evidence scope, and difficulty) alongside gold sub-question decompositions (query rewrite) for 69% of records, enabling performance to be disaggregated along fine-grained dimensions obscured by aggregate metrics."

  3. Reference baselines: covering sparse (TF-IDF, BM25), dense (bi-encoder), and cross-encoder architectures, demonstrating through hard-versus-random negative contrasts that curated distractors degrade model accuracy by 13.0–20.5 percentage points.

  4. Transparency and extensibility: "documented construction criteria, a deterministic label-normalization and repair pipeline with a machine-readable change log, per-passage identifiers and text hashes, a hard-negative taxonomy, and a target-versus-realized distributional audit."

The hard negatives (6,021 total) are categorized by the relationship between the negative's originating filing and the source record: 79.9% (n=4,835) are Same industry, different company, 13.2% (n=796) are Same company, different year/form, 6.9% (n=420) are "Same company, same year & form", and <0.1% (n=1) are cross-industry. The authors note: "Because hard negatives were chosen by a human reader searching for the most confusable evidence rather than sampled by a retrieval heuristic, they concentrate in the 'same industry, different company' bucket... This distribution is a deliberate annotation choice, and it is precisely the configuration that most often confuses lexical and embedding-based retrievers."

An audit identified 11 hard negatives that duplicated a supporting passage of their own record (removed), and 442 hard negatives (7.3%) that coincide with a supporting passage of a different record, where the passage may legitimately be relevant to more than one question — these are flagged in a released taxonomy file so users can filter them.

FinRank supports five tasks: passage retrieval, reranking with hard negatives, multi-passage reasoning, answer generation, and query rewriting. Two evaluation regimes are defined:

  • Global pooled corpus C: the union of every non-empty supporting-passage text and hard-negative text appearing anywhere in FinRank, deduplicated on the raw passage text — C = 5,230 unique passages. The authors emphasize: "because C consists solely of annotator-selected positives and curated distractors, retrieval over C is a curated passage-ranking stress test with an elevated density of confusable passages, not retrieval through the complete text of each filing."

  • In-record candidate set Lr: Lr = Gr ∪ HNr is the union of r's supporting passages and its hard negatives, used for reranking and hard-negative-discrimination evaluations.

Seven systems were evaluated: TF-IDF, BM25 (BM25Okapi), a dense retriever (all-mpnet-base-v2), a cross-encoder reranker (ms-marco-MiniLM-L-6-v2), a stronger general-purpose embedder (bge-large-en-v1.5), a finance-adapted embedder (FinLang/finance-embeddings-investopedia), and a 7B instruction-tuned embedder (e5-mistral-7b-instruct), all under uniform 512-token truncation. A metadata-filtered BM25 baseline restricted to corpus entries matching the query record's source filing was also reported.

Results show the difficulty of the setting:

Model Recall@1 Recall@5 Recall@10 Recall@20 MRR nDCG@10

TF-IDF 7.7 23.7 32.8 41.9 23.9 21.9

BM25 8.3 24.5 32.1 40.1 23.8 21.8

Dense (all-mpnet-base-v2) 6.5 21.5 31.8 42.0 22.2 20.4

  • CE rerank of mpnet top-20 9.6 27.5 35.2 42.0 26.9 24.6

Dense (finance-adapted BGE) 5.2 16.5 22.4 29.8 16.5 14.8

Dense (bge-large-en-v1.5) 8.8 25.9 35.6 44.6 25.9 23.8

Dense (e5-mistral-7b-instruct) 10.1 33.1 44.8 55.0 31.0 29.7

The abstract summarizes: even a 7B instruction-tuned embedder reaches only 44.8% Recall@10 on the pooled evidence corpus; sub-billion-parameter encoders gain at most 3.5 points over BM25, a finance-adapted embedder trails BM25 by 9.7 points.

Restricting the corpus to entries matching the query record's filing (ticker, year, doc type) eliminates 92.9% of that record's hard negatives and shrinks the pool to ≈173 candidates on average, lifting BM25 from 32.1 to 55.0 Recall@10 (MRR 23.8 → 41.9). The filter excludes at least one gold passage for 33 records and every gold passage for 2. The authors note this is genuinely powerful and deployed filing assistants should use it, but even under this filter, nearly half of the gold evidence is missed at k = 10.

On Lr, MRR values ranged from 72.3 (finance-adapted BGE) to 83.6 (e5-mistral-7b-instruct), with nDCG@5 from 70.2 to 80.5.

Pairwise accuracy across (positive, hard-negative) pairs versus (positive, random negative) pairs shows the hardness premium:

  • TF-IDF: 70.3 vs. 91.9 (gap +21.6, reported as 19.3 in one framing — the paper's figure shows +19.3, +17.5, +16.0, +15.9, +13.0, +14.2, +20.5 across models)

  • BM25: 74.8 vs. 87.8

  • Dense (all-mpnet): 74.5 vs. 93.7

  • Fin-BGE: 76.3 vs. 93.7 (gap +17.5)

  • bge-large: 80.2 vs. 96.2 (gap +16.0)

  • e5-7B: 76.1 vs. 92.0 (gap +15.9)

  • Cross-encoder: 77.8 vs. 90.8 (gap +13.0)

The paper states: "Every model ranks a positive above a random negative 88–96% of the time but above a curated hard negative only 70–80% of the time, a 13.0–20.5 point drop. This gap is the empirical hardness premium of the released hard negatives, and it is the single result that most directly justifies shipping them as a first-class benchmark asset."

On the 819 records with non-empty query rewrites, concatenating rewrites with the question improves Recall@10: TF-IDF 32.3→41.1, BM25 29.9→38.7, Dense 30.7→34.1. The authors note this should be read as an oracle upper bound on decomposition-assisted retrieval since rewrites are annotator-authored with gold evidence in view.

10-Q, multi-passage, and quantitative records score lowest. For example, BM25 Recall@10: 10-K 33.9 vs. 10-Q 18.0; qualitative 34.7 vs. quantitative 26.5; single-passage 44.7, two-passage 28.0, multi-passage 21.5; Easy 37.6, Medium 32.4, Hard 25.9.

The authors draw four key observations: (1) ranking within a curated pool of confusable passages is hard, with more than half of gold evidence outside the top ten even for the strongest model; (2) encoder capacity dominates domain labels on this heavily templated text, with a finance-adapted embedder trailing BM25 by 9.7 points while the 7B instruction-tuned embedder gains 12.7 points; (3) the hardness premium justifies curated hard negatives; (4) aggregate scores hide weakness in 10-Q, multi-passage, and quantitative strata. They caution that the 10-Q gap should be read descriptively rather than causally: all 10-Q records come from the oil and gas sector and predominantly from 2025, so document type is confounded with sector, filing year, and annotator.

The authors acknowledge: small scale and skew (1,185 records, 22 companies, 3 sectors; 75% from 2025; 87% 10-K; roughly 2:1 qualitative-to-quantitative); single-annotator review with no formal inter-annotator-agreement statistic (Correctness on FinRank is thus single-annotator with sampled review rather than consensus-adjudicated); label inconsistencies in raw data resolved by normalization; 442 hard negatives overlapping supporting passages of other records; tables and figures represented as text only; and generative answer-quality results left to future work.

FinRank is derived from public SEC filings, contains no non-public personal information, and is released under CC BY-NC 4.0 for non-commercial research. The authors state: Strong performance on FinRank does not establish suitability for investment decisions, valuation, or trading, and FinRank-aligned systems should complement rather than replace qualified human analysts. The dataset is not intended for generating automated investment advice, valuations, or trading signals.

The paper concludes: "FinRank is complementary [to FinCARDS] on both counts: it supplies the annotated positives and curated distractors that such constraint-aware rerankers are scored against, and its hard negatives are drawn predominantly from other filings, targeting precisely the cross-document setting left open there. Future work includes broadening coverage to more sectors, companies, and filing years; a stratified double-annotation study with reported inter-annotator agreement; executing the generation and RAG protocols with LLM-based systems; and extending the schema and hard-negative construction to multimodal (tabular and graphical) evidence."

Improvements for AI systems

Based on FinRank, I would make the following concrete improvements to AI systems for financial QA and retrieval:

  • The single clearest signal is the “hardness premium”: models rank positives above random negatives 88–96% of the time, but only 70–80% above curated confusable negatives — a 13–20.5 point drop.

  • Improvement: augment training and fine-tuning with hard negatives dominated by near-identical boilerplate from other firms in the same sector, rather than random negatives from the same filing.

  • The improved system can: suppress a competitor’s risk-factor paragraph, revenue-recognition note, or accounting-policy disclosure that is lexically near-identical but legally irrelevant to the target firm.

  • Metadata-filtered BM25 (ticker + year + doc type) eliminated 92.9% of hard negatives and lifted Recall@10 from 32.1 to 55.0 with MRR from 23.8 to 41.9.

  • Improvement: build a two-stage retriever that first restricts candidates to the query’s source-filing metadata (company, filing year, form type) and only then applies semantic ranking, while explicitly handling cases where gold evidence crosses periods or forms.

  • The improved system can: avoid matching a 2024 10-K passage when the query targets the 2025 10-Q, and reduce spurious cross-filing confusions before neural reranking is used.

  • Concatenating annotator-authored query rewrites lifted TF-IDF Recall@10 from 32.3 to 41.1, BM25 from 29.9 to 38.7, and dense retrieval from 30.7 to 34.1.

  • Improvement: train a lightweight decomposition model that rewrites a financial question into evidence-scoped sub-queries (e.g., isolating “what is the tax rate” into “effective tax rate footnote,” “reconciliation of effective rate,” “variance explanation”), and retrieve with each sub-query separately before fusing results.

  • The improved system can: handle multi-passage and two-passage evidence scope questions (which scored lowest, e.g. BM25 Recall@10 21.5 for multi-passage) by matching each sub-question against a distinct supporting passage.

  • The finance-adapted embedder (FinLang/finance-embeddings-investopedia) trailed BM25 by 9.7 points, while the 7B instruction-tuned e5-mistral-7b-instruct gained 12.7 points over BM25.

  • Improvement: prioritize instruction-tuned, higher-capacity encoders (7B class) and prompt-based embedding formulations over smaller finance-specific models for templated regulatory text. Use FinRank to test whether a “domain label” is actually helping or hurting.

  • The improved system can: achieve 44.8% Recall@10 on a deliberately confusable pooled corpus, where smaller encoders plateau near 32–36%.

  • FinRank’s core claim: a numerically correct answer can be grounded in the wrong evidence because templates are near-identical across firms and periods.

  • Improvement: require the generation pipeline to output a cited passage ID (or page number) alongside each answer, and score answer correctness only when the provenance passes a hard-negative discriminator. Train a reranker to distinguish gold evidence from the 79.9% of hard negatives coming from same-industry different-company filings.

  • The improved system can: answer “What was the company’s revenue?” with the correct firm’s revenue note, not a competitor’s nearly identical note, and flag when no passage in the target filing supports the composed answer.

  • The paper shows 10-Q Recall@10 is 18.0 vs 33.9 for 10-K; quantitative questions score 26.5 vs 34.7 for qualitative; multi-passage 21.5 vs 44.7 for single-passage. Aggregates hide these.

  • Improvement: report and optimize on stratified slices — especially for 10-Q filings, quantitative reasoning questions, and multi-passage evidence questions — rather than aggregate recall/MRR alone.

  • The improved system can: identify and close the “difficult strata” gap, such as locating quantitative evidence spread across two or more footnotes, or handling the shorter, more variable 10-Q disclosure style.

  • Cross-encoder reranking of the dense top-20 improved Recall@10 from 31.8 to 35.2, but pairwise positive-vs-hard-negative accuracy was only 77.8% vs 90.8% for random negatives.

  • Improvement: fine-tune rerankers with a ranking loss that explicitly pairs each gold passage against the record’s hand-curated hard negatives, using the released hard-negative taxonomy to weight confusable types (same-industry > same-company-different-year > same-filing).

  • The improved system can: reorder evidence such that the gold footnote appears in the top 5 even when 4 near-identical competitor footnotes are present, directly improving downstream answer provenance.

  • 442 hard negatives (7.3%) coincide with supporting passages of other records, and 11 duplicates within a record were removed.

  • Improvement: when building retrieval training sets, apply deterministic deduplication and cross-record overlap filtering so that a passage which legitimately answers multiple questions is not incorrectly treated as a hard negative.

  • The improved system can: avoid penalizing legitimate multi-passage reuse and produce cleaner learning signals, reducing false negatives during fine-tuning.

Abstract

Financial question answering is typically evaluated by answer correctness, yet in SEC filings a plausible and even numerically correct answer can be grounded in the wrong evidence. Similar facts and disclosures recur across sections of a filing, across reporting periods of the same firm, and across comparable firms. FinRank targets this provenance-sensitive retrieval problem by requiring systems to identify evidence for the intended entity, reporting period, and disclosure context. The benchmark contains 1185 manually authored question-answer records over the 10-K and 10-Q filings of 22 companies. Each record includes a reference answer, gold supporting passages, and hand-curated hard negatives drawn from confusable passages within filings, across reporting periods, and across comparable firms. FinRank evaluates passage retrieval, reranking, and hard-negative discrimination as separately measured tasks. Baseline results demonstrate the difficulty of this setting: among the evaluated systems, even a 7B instruction-tuned embedder reaches only 44.8% Recall@10 on the pooled evidence corpus; sub-billion-parameter encoders gain at most 3.5 points over BM25, a finance-adapted embedder trails BM25 by 9.7 points, and pairwise accuracy falls by 13.0-20.5 percentage points when random negatives are replaced with the curated hard negatives. FinRank provides an evidence-first benchmark for developing financial question answering systems that are not only accurate but also grounded in the correct disclosure.

Sources

Related papers