summary
The episode reviews FinRank, a benchmark for financial question answering over SEC filings that prioritizes evidence grounding over answer correctness. Hosts discuss how systems often retrieve correct-looking answers from wrong filings, cite results like BM25 improving from 32% to 55% recall with metadata filtering, and emphasize provenance as critical for audit and compliance.
Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Next we'll be talking about the paper "FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings".
Jane: The paper was written by Sasan Mansouri, Daniel Saad, Mark Wahrenburg, Manu Weissel and Fabian Woebbeking from University of Groningen and Goethe University Frankfurt and DataNXT GmbH and Halle Institute for Economic Research and Martin Luther University Halle-Wittenberg.
Tom: Stay tuned as we take you through the paper and discuss its implications.
Title: Tom: Welcome back, everyone. Today's paper is called "FinRank: An Evidence-Grounded Benchmark for Financial Question Answering and Retrieval over SEC Filings," and it comes from Sasan Mansouri at the University of Groningen, Daniel Saad, Mark Wahrenburg, and Manu Weissel at Goethe University Frankfurt, and Fabian Woebbeking at the Halle Institute for Economic Research and Martin Luther University Halle-Wittenberg. It's a benchmark for financial QA over SEC filings, and the emphasis on evidence grounding in the title is really the whole story.
Jane: I'm glad you read out the full author list, because DataNXT, a Frankfurt-based company, is on it too. That mix of university research and a firm that builds financial eye tools shows up all over the design. This feels built for analyst workflows rather than for a leaderboard.
Tom: The core problem is that in SEC filings a plausible answer can be grounded in the wrong evidence. The authors argue the primary bottleneck in automated financial analysis is evidence discrimination rather than answer composition — telling apart the target firm's disclosure from a competitor's near-identical boilerplate, or from a prior period's filing. In practice, that wrong evidence is a passage that looks correct at first glance.
Jane: That rings true. Risk-factor sections in 10-Ks are heavily templated, so two pharmaceutical companies can describe their litigation exposure in almost the same words. If a system can't tie the answer to the correct filing, the answer is worthless even when it's numerically correct.
Lu: The paper spans three sectors — pharmaceuticals, oil and gas, and automotive — across 22 companies and filings from 2024 and 2025. Those confusable passages aren't hypothetical. They're real competitor filings sitting right next to the correct evidence in the same corpus.
Meng: And the authors flip the usual priority. General-domain QA treats supporting evidence as a secondary annotation, but here an answer's validity depends on strict provenance to the underlying disclosure. For an auditor or a compliance reviewer, that distinction is what separates useful output from a liability. The paper's whole design principle is that you can't evaluate financial answers without knowing where they come from.
Lalam: That's the broader significance, from where I sit. Regulated finance cares about provenance as much as correctness, and once you put eye into audit or compliance, a citation pointing at the wrong company can undermine the whole output and the trust placed in it, even if the answer reads perfectly. That's why this evidence-first framing matters beyond any single benchmark.
Tom: So the team built a benchmark directly from that pain point. There are 1185 manually written questions, and each one carries gold evidence plus a set of deliberately confusing distractors. Let's look at what is actually inside the dataset.
Summary: Tom: We've set the stage — the authors built this dataset around the evidence problem. Now, what's actually in it? There are 1185 question–answer records, all manually authored over the 10-K and 10-Q filings of 22 companies. Five business students did the writing: they read the filings, wrote every question, wrote every reference answer, and transcribed the supporting passages by hand, with page references.
Jane: And they didn't use a language model for any of that, which is a meaningful design choice
Paper discussion segment 3: Tom: So to recap, FinRank is a benchmark that checks whether financial QA systems can find the right evidence in SEC filings, not just the right answers.
Jane: And that focus on evidence points to a few concrete improvements the authors want the field to adopt. The biggest one is simple: stop grading financial answers without checking where the evidence came from.
Tom: Right, because a citation pointing to the wrong company's filing is worse than no citation at all in audit or compliance work. The paper basically argues that provenance should be a first-class metric.
Jane: They also suggest practical fixes, not just evaluation ones. The metadata filter is a good example — when the system knows which ticker, year, and form it should be looking at, BM25 jumps from 32 to 55 percent recall at ten.
Tom: That's huge, and it tells you that deployed tools should use filing metadata aggressively before doing any semantic search. It's cheap, deterministic, and it eliminates most of the confusable distractors upfront.
Jane: And for the harder cases that survive filtering, the paper recommends stratified reporting. Aggregate scores hide the fact that multi-passage and 10-Q questions are much harder across the board.
Tom: That's an important implication for anyone building on this — if you only look at your mean Recall@10, you'll miss that your model collapses on quarterly filings or multi-source synthesis.
Jane: The implications go beyond model development, too. In regulated settings, an answer with the right citation is actionable; an answer with the wrong citation is a liability. FinRank makes that trade-off measurable.
Tom: Which could push vendors of financial eye tools to actually show evidence-attribution numbers in their marketing, instead of just answer accuracy.
Jane: Exactly. And the authors leave the door open for more: they haven't run generative RAG baselines yet, and they explicitly call for a double-annotation study to verify label correctness.
Tom: So the natural next step for the benchmark is human adjudication at scale. That's the kind of work that would make FinRank even harder to dismiss in high-stakes deployment.
Jane: And we'll get into what that future work might look like, and whether this benchmark can hold up outside the three sectors it covers, right after the break.
Paper discussion segment 4: Tom: So far we've covered the dataset's design and the changes the authors want the field to adopt — now let's look at how they open the paper, because the abstract does some clever framing before you even get to the numbers.
Jane: It does. The opening sentence sets up a contrast: most financial QA benchmarks grade answer correctness, but FinRank grades whether a correct-looking answer is grounded in the right evidence. That immediately shifts the goal posts.
Tom: They call that "evidence discrimination" — the primary bottleneck in automated financial analysis. Not answer composition, but telling apart the target firm's disclosure from a competitor's near-identical boilerplate. That idea runs through the whole abstract.
Jane: And they argue this is what makes financial QA fundamentally different from open-domain QA. In general search, there's usually one obvious source for a fact. In SEC filings, the same standard accounting language appears across companies and across reporting periods.
Tom: Right, that's why they also flag document length and structural complexity. A single 10-K spans hundreds of pages, mixing accounting, legal, and forward-looking language — and answers often depend on evidence scattered across tables, footnotes, and narrative sections. That complicates retrieval in a way open-domain benchmarks don't capture.
Jane: The abstract also telegraphs the headline results. Even a 7B instruction-tuned embedder reaches only 44 point 8 percent recall at ten on the pooled evidence set. That's a strong statement about how hard this benchmark actually is.
Tom: And the smaller models barely beat BM25 — at most 3 point 5 points of Recall@10 — while a finance-adapted embedder trails BM25 by almost ten points. The authors are making the point that domain labels on embeddings don't automatically transfer to templated filing text.
Jane: That result should make anyone building finance-specific encoders sit up. It suggests the real gains come from scale or from adapting to SEC filings specifically, not just any financial corpus.
Tom: Exactly. And the abstract ends with the hard-negative finding: pairwise accuracy drops 13 to 20 points when you replace random negatives with the curated ones. That's the empirical proof that these distractors are genuinely confusable.
Jane: Which brings us to how those hard negatives were actually constructed and why the "same industry, different company" bucket dominates — we'll dig into that next.
Conclusion: Tom: So today we've been talking about FinRank, a benchmark that grades financial QA systems on whether they find the right evidence in SEC filings rather than just the right-looking answer.
Jane: And honestly, the most striking part is how hard even strong systems find that task. The best model in the paper, a 7B instruction-tuned embedder, only reaches about 45 percent recall at ten on the pooled evidence set.
Tom: That number should surprise people who think retrieval is basically solved. And it gets worse — a finance-adapted embedding model actually trails plain BM25 by nearly ten points. So throwing domain labels at the problem doesn't automatically help on templated filing text.
Jane: The authors show that knowing the target filing's metadata — ticker, year, form type — is far more powerful than any semantic trick. Filtering to the right filing lifts BM25 from 32 to 55 percent recall at ten, which is a huge jump for zero model changes.
Tom: But even with that filter, about half the gold evidence is still missed. That tells you the remaining challenge is genuine semantic discrimination between near-identical disclosures, not just corpus size.
Jane: And that's where the hard negatives come in. Every model drops 13 to 20 points of pairwise accuracy when you swap random distractors for human-curated ones. Those numbers validate the whole dataset design.
Tom: The authors are also upfront about limits. The benchmark is small, skewed toward 10-Ks and qualitative questions, and single-annotator without formal agreement statistics. That's why they push stratified reporting so hard.
Jane: Their roadmap is clear: run actual RAG generation baselines, do a double-annotation audit, and broaden coverage beyond three sectors. All of that would make FinRank even more credible for high-stakes use.
Tom: And for now, the practical message for anyone building financial assistants is simple — check your evidence, not just your answer. FinRank gives you the tool to do exactly that.
Jane: Great note to end on. Next up, we're switching gears to a paper that tries to bypass document retrieval altogether by feeding models curated vendor data directly through the Model Context Protocol. We'll see whether that shortcut holds up.