2608.09393-Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law".

Jane: The paper was written by Rose Cymbler, Daniel Guez and Laurent Fabre from Talia and Databricks.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We've both read through this one, and honestly it hasn't left my head since. Let me get the whole thing on the table, because the finding deserves a proper airing.

Jane: The core claim is simple to state. The authors looked at French tax law, which gets amended every single year through finance laws, and asked whether large language models can answer questions about what the law said at a specific past date. Short answer: no.

Lu: They tested eleven models, including the biggest frontier systems, and from memory alone the models scored three percent mean strict accuracy. Three percent, on questions about things like tax rates that were in force decades ago.

Meng: But the number that grabbed me is underneath that. They also ran a standard RAG setup over the current version of the tax code — which is what most deployed legal products actually do — and it got two point seven percent. Statistically indistinguishable from answering with no retrieval at all.

Jane: And it did so confidently. The system retrieved a real article, quoted a real rate, and the rate was simply wrong for the date in the question. That's what the paper names temporal misgrounding: grounding an answer in something real but inapplicable.

Tom: They built a versioned corpus to study this — thirty-two thousand four hundred thirty-six article-versions of the French tax code, spanning ninety-three years — and then they conditioned retrieval on the date in the query.

Lalam: And the payoff is dramatic. The same models, answering the same questions, jump to ninety-eight point three percent mean strict accuracy once the retriever knows about versions and dates. The bottleneck was never the model. It was the missing temporal dimension in the retriever.

Lu: With an oracle supplying the right article, it hits ninety-nine point one. So version selection was doing almost all of the work.

Meng: And the evaluation design is just as important. They refuse to use an LLM as judge, because a judge that shares the same recency bias will bless a fluent answer quoting the wrong-date value. Every response is scored against atomic nuggets — the right article number, the right numeric value — checked by regex and numeric tolerance.

Jane: So we've got a named failure mode, a benchmark that measures it, and a retrieval method that fixes it. For anyone building legal eye products, this is the difference between a tool that sounds confident and a tool that's actually right.

Tom: And the paper opens by arguing this failure is structural, not anecdotal. Let's walk through page one.

Page 1 of the paper: Jane: So we've got the whole picture sketched out. Page one starts with a single, very concrete example. A taxpayer asks what the standard corporate income tax rate was in 2018. The right answer is thirty-three and a third percent. The current rate is twenty-five percent. And a frontier model trained through 2025 will just say twenty-five.

Tom: Because that's what it knows. That's the phenomenon they name temporal misgrounding, and they break its causes into three. The first is parametric recency bias — training data overrepresents the most recent legal state, so the model's priors pull toward current law.

Lu: The second is that standard dense retrievers index by semantic similarity and never condition on the date implicit in the query. You can write "in 2018" into the question and the retriever treats it as decoration.

Meng: And the third is the sneakiest. Article-number aliasing. Article 219 of the French tax code exists in every version of the code — there was an article 219 in 1950 and in 2026. The identifier is stable, but the content changed. So naive retrieval can't even tell the versions apart.

Tom: Then they frame the research question: how much does temporal misgrounding actually bottleneck state-of-the-art LLMs and RAG systems? And they deliberately isolate the regime where the date-applicable answer differs from current law, because that's where the failure is diagnostic.

Lu: That's a deliberate scoping choice. The benchmark isn't measuring average legal QA performance. It's measuring the failure mode at its worst, where temporal drift is the entire game.

Jane: And the contributions are all listed there — a characterization of the failure mode with a taxonomy, the versioned corpus, the benchmark, and a controlled three-condition experiment.

Lalam: Page one also states the paper's thesis in one line: legal question answering should be treated as a temporally-indexed retrieval problem, where the date is a first-class part of the query rather than a side detail.

Tom: And from there the paper positions itself against everything that came before. Page two pulls in the surrounding literature, and there's a telling contrast with what the existing benchmarks don't do.

Page 2 of the paper: Tom: So the phenomenon and the diagnosis are on the table. Page two is the literature review, and it's doing real work. The authors go through the major legal NLP benchmarks — LegalBench, LEXTREME, LEXam — and the pattern is stark. They all treat the law as a fixed snapshot.

Jane: LegalBench has a hundred sixty-two tasks, all in English, with no temporal indexing to speak of. LEXam evaluates reasoning on a fixed legal state. So the date dimension is just absent from how these benchmarks think about legal QA.

Lu: The closest ancestor is OfficeQA Pro — an enterprise benchmark over U.S. Treasury Bulletins, a hundred thirty-three questions across eighty-nine thousand pages spanning nearly a century.

Meng: And it already showed frontier models below five percent on parametric knowledge alone. With direct corpus access they reached thirty-four percent. So even with retrieval, there was enormous headroom.

Tom: This paper takes that methodology and moves it to a setting where documents are revised in place — the same article number means different things at different times. That's the structural difference.

Jane: They also position against a graph-based approach called SAT-Graph RAG, which resolves point-in-time queries through an ontology. The authors' pitch is that you don't need any ontology at all — just explicit version indexing and date-conditioned retrieval over the raw text.

Lalam: And they cite two concurrent works that independently found the same failure — one German study with three hundred twelve statutory QA pairs, another on training-cutoff bias in legal search agents. That convergence is actually reassuring. When three groups hit the same wall at the same time, the wall is real.

Lu: But there's a sharp methodological divergence. The German study scores with an LLM-as-judge. This paper refuses, because a judge that shares the recency bias will validate answers that quote the wrong-date value. That's the circularity they're determined to break.

Meng: So the differences come down to three things: deterministic scoring, a corpus version-indexed at fine granularity with future-effective versions included, and a hardness filter across eleven models rather than just frontier ones.

Tom: Which sets the stage for the core argument of page three: why static retrieval is doomed against this kind of corpus in the first place.

Page 3 of the paper: Jane: We've seen where prior benchmarks fall short. Page three is where the paper earns its keep — it explains structurally why static RAG fails, and it starts with three properties of legal corpora that are beautifully concrete.

Tom: Property one: same identifier, different content. Article 219 carried the standard corporate rate, and that rate moved — thirty-three and a third percent in 2018, thirty-one in 2019, twenty-eight in 2020, twenty-six and a half in 2021, then twenty-five from 2022 onward. Same article number the whole way through.

Lu: Property two: date-dependent correctness. Each version carries an explicit start date and end date, so a question anchored in 2018 has exactly one correct version. There's no approximately-right version. A nearby version is just wrong.

Meng: And property three is the kicker for retrieval. Successive versions of an article are textually near-identical — sometimes they differ by a single rate or threshold. So their dense embeddings are each other's nearest neighbors. Semantic similarity alone cannot disambiguate them.

Jane: From those three properties they derive four failure modes. The dominant one is current-law substitution — the retriever returns the in-force version regardless of the question's date.

Tom: The others are future-law leakage, where a not-yet-in-force version surfaces in answer to a present-tense question; wrong-amendment resolution, where asking for the version before a specific law yields something arbitrary; and multi-version confusion, where a before-and-after comparison collapses to the single highest-scoring version.

Lu: Then they take on the obvious objection: wouldn't a bigger, newer model just memorize the history? Their answer is no, for structural reasons. Training data skews toward recent, widely-cited content. The historical volume is enormous — over thirty thousand versions across just two codes. And memorization wouldn't fix disambiguation anyway, because of property three.

Meng: There's a sharp line in there — that temporally-grounded questions are easy to recognize but hard to answer without a version-indexed corpus. Recognition isn't the bottleneck.

Lalam: So the paper's claim is that fixing this requires structural changes to retrieval itself — version indexing and date conditioning — rather than bigger models. And that's exactly what they go build in the next two sections.

Tom: Right, they start building it on page four, where the corpus construction is laid out in detail.

Page 4 of the paper: Tom: So the failure is structural, and the answer has to be structural too. Page four is pure construction — it's where they build the versioned corpus, pulling from the official Légifrance API in France.

Jane: The key endpoint returns an article's complete version history, not just the current text. Each version carries a distinct identifier under a stable article identifier. So the article is constant, and every historical modification is its own retrievable object.

Lu: The numbers are striking. Thirty-two thousand four hundred thirty-six article-versions across six tax codes. The main code alone has twenty-one thousand versions over thirty-seven hundred articles — an average of five point seven versions per article.

Meng: And the record holder is article 81, on tax-exempt income, with ninety-four distinct versions. Ninety-four edits over the decades, mostly from annual finance laws.

Jane: The temporal span runs ninety-three years, from 1938 to 2031. That future endpoint matters — some laws are already enacted with entry into force deferred, so the corpus holds versions that are legislated but not yet in effect.

Tom: They also built a secondary resource: linking court decisions to the specific article version that applied at the decision date. A regex extractor with a proximity veto and a fiscal-context filter produces sixty-nine thousand two hundred eight version-aware links across thirty-two thousand decisions, with ninety-eight to ninety-nine percent precision and decision-level recall in the eighties to nineties on the jurisdictional sources.

Lu: That's auxiliary — it doesn't feed the controlled experiment — but it's the infrastructure for the citation and synthesis tracks.

Meng: And then the benchmark itself. Four regimes. R1 is citation extraction, R2 is deterministic computation, R3 is the temporal reasoning track, the core of the paper, and R4 is multi-document synthesis.

Tom: R3 has two hundred nine scored questions across thirty-three articles, out of two hundred twenty-one released — twelve get flagged out of the answerable scope. Every question was reviewed by a French tax professional, and the correct value is anchored to a specific version.

Jane: So the corpus is built and the benchmark structure is set. The scoring methodology comes next, and that's page five — where they make the controversial call about never letting an LLM be the judge.

Page 5 of the paper: Tom: Corpus and benchmark structure are done. Page five gets into the evaluation methodology, and this is where the paper makes a really deliberate stand. They score answers against atomic ground-truth nuggets — an article identifier matched by regex, a numeric value matched with tolerance — and they never use an LLM to judge.

Jane: Their argument is tight. Temporal grounding is exactly the axis where LLMs share a systematic bias. So a model judge inherits the same recency bias and will accept a fluent answer quoting the wrong-date value, because that value matches the judge's own prior.

Lu: There's a leakage problem they're honest about too. The article-number nugget is given away by the question text itself for almost eighty-four percent of the questions. So they also track value-only coverage, and the headline effect is carried entirely by the date-anchored numeric value.

Meng: Then there's the parametric knowledge filter. Every candidate question is probed against all eleven models in four sampling draws, and any question where a model produced the gold value is dropped. That's how the set becomes all-model-hard — nobody gets in from memory.

Tom: And a second filter enforces the experimental premise: the gold value has to be absent from the current in-force version of the article. For two hundred eight of the two hundred nine scored questions, the current text simply doesn't contain the date-applicable value.

Jane: The single exception is kept deliberately as a control — and notably, the static RAG baseline gets that one right later. So the baseline's failure on the rest is version drift, not a broken pipeline.

Tom: The curation pipeline mixes automation with human work. A factory script scans version histories for value transitions — a rate or threshold changing between consecutive versions — and surfaces them as candidates. The authors write the questions, verify every answer against the corpus, and a qualified French tax professional reviews the whole set.

Lalam: And the targeting is smart. They deliberately avoid headline rates like the corporate rate or the income tax scale that frontier models memorize, and instead go after obscure, non-rounded parameters — indexed allowances, thresholds, per-installation tariffs. That keeps the benchmark genuinely hard.

Lu: And of the two hundred twenty-one released questions, twelve get flagged out of the answerable scope — four because the value is annually indexed by INSEE and published administratively, eight caught in review as curation errors or ill-posed under their date anchor.

Jane: So by the end of page five we have a frozen, filtered, human-verified test set. Page six shows how they run the controlled experiment on it.

Page 6 of the paper: Tom: So we've got a frozen, filtered, human-verified test set. Page six sets up the controlled experiment around it, and every layer of the design is deliberate.

Jane: The scored set is frozen before any retriever development happens — there's a file called killer qids that pins the questions down — and the reranker they later tried was only trained on articles disjoint from the benchmark set.

Lu: The scale story is interesting too. The original track had thirty-five questions, and they expanded it to two hundred twenty-one in response to reviewer feedback, to make room for eleven models and proper statistics.

Meng: There's also an honest accounting of residual leakage. Even after the filter, at evaluation time at least one model produced the gold value for thirty-seven of the two hundred nine questions — about eighteen percent. GPT-5 point 5 did it twenty-five times. The filter guarantees hardness at selection time, not forever after.

Tom: Then the three conditions. Condition A: the model answers from parametric knowledge alone — no retrieval, no web. Condition B: RAG over a current-version-only corpus, and here's the charitable detail — the system is handed the correct article identifier. Only the current version is exposed.

Jane: Condition C is the versioned setup, split in two. Cor is the oracle version selection — gold article, so you isolate version selection alone. Cprod is the end-to-end retriever, which must find both the article and the version with no oracle. It combines a domain-adapted dense encoder with BM25, fused by reciprocal rank fusion, over three representative versions per article, feeding the top five to the model.

Lalam: The model lineup spans the frontier — Claude Opus 4 point 7 and 4 point 8, Sonnet 4 point 6, GPT-5 point 4 and 5 point 5, with Gemini 2 point 5 Pro standing in for a rate-limited Gemini 3 — plus five open-weight systems like Llama 4 Maverick and Gemma 3 27B.

Meng: Qwen actually got swapped between filtering and evaluation. The 72B model used at selection time was retired from serverless inference, so they substituted the larger Qwen 3 235B — and that replacement scored zero on condition A, so the hardness probe survived.

Tom: Each condition maps to a falsifiable hypothesis. H1 says condition A is uniformly low across scale and provider. H2 says B retrieves the applicable version zero percent of the time and stays under ten percent strict — with the caveat that the low strict score is partly constructed by the divergence filter, so the real falsifiable content is the zero provenance.

Jane: H3 says the oracle closes most of the gap — above eighty percent strict, a hundred percent provenance. And H4 says the realistic retriever recovers essentially the oracle ceiling, with any residual gap living in first-stage recall.

Lu: So the hypotheses are sharp, and each condition tests a different layer of the pipeline. Page seven delivers the results.

Page 7 of the paper: Tom: The experiment design is settled, with three conditions and four hypotheses. Page seven opens with a check on the corpus itself, and the numbers are worth sitting with. Across the whole corpus there's a mean of just over four versions per article, but the distribution is wild — the main tax code alone averages five point seven versions per article, and article 81 has ninety-four.

Jane: Then the results. Condition A — pure parametric knowledge — lands at three point zero percent mean strict accuracy across the eleven models. The confidence interval runs from one point four to four point seven. It's a wall.

Lu: And condition B, static RAG, does not improve on it: two point seven percent, statistically indistinguishable. The system is handed the correct article, reads the current version, and still can't answer questions about past dates.

Meng: But we should push on that number a little, because the paper admits the low strict score is partly by construction. The divergence filter dropped every question where the current text still contains the gold value. So B failing on strict accuracy is almost predetermined.

Tom: That's fair, and they say it themselves. The falsifiable content of that hypothesis is the zero provenance — a single-version index structurally cannot hold the applicable version — and the ceiling below ten percent. The control condition is what shows the failure isn't an artifact of bad questions.

Jane: And the provenance result is the one that should worry anyone building legal products. The static retrievers found the date-applicable version zero percent of the time. Not one percent, zero. And the paper's phrase is striking — it fails not silently but confidently, grounding on a real, well-formed, but inapplicable version.

Lalam: The control condition does a lot of work. Same models, same prompts, same pipeline — but with the oracle serving the date-applicable version of the same article, the numbers jump to ninety-nine point one percent strict. The questions are answerable, and the models extract the values perfectly when the right text is in front of them.

Tom: And the one question where the current text still contains the gold value — static RAG gets that one right. That seals the diagnosis: the entire deficit is version drift.

Jane: So the static baseline fails on both counts, oracle selection succeeds, and the real question becomes whether a realistic retriever can reach that result without being handed the answer. Page eight answers that.

Page 8 of the paper: Jane: So the static baseline has failed on both counts. Page eight delivers the operative result — the end-to-end retriever, with no oracle, has to find the article and the version on its own. It reaches ninety-eight point three percent mean strict accuracy.

Tom: Every single one of the eleven models crosses ninety-five percent. And the date-applicable version lands in the retrieved top five ninety-nine percent of the time, which is a complete inversion of the static baseline's zero.

Lu: The oracle-article ablation sits at ninety-nine point one, so the gap between the realistic retriever and the ceiling is only eight tenths of a point. And the paper locates that residual precisely: it's first-stage recall. Two questions on one article — article 1417 — where the correct version fell outside the retrieved top five.

Meng: A cross-encoder reranker adds nothing on this set. So the lever is recall, not top-one reranking. Once the corpus is versioned, date resolution is essentially solved; finding the right article is what's left.

Jane: Then there's the statistics, and they're careful about it. The questions cluster by article — thirty-three clusters, one article carrying thirty-four questions alone — so they bootstrap by resampling articles, with the model as the unit of inference. Ten thousand iterations.

Tom: And the per-model McNemar tests are decisive. Every one of the eleven models shows the gain, with p-values below ten to the minus fifty-five. So the conclusion doesn't depend on pooling all models together.

Lu: There's also a leave-one-article-out analysis. The pooled result barely moves — from ninety-eight point one when you drop one article to ninety-nine point two when you drop another. No single article is carrying the win.

Meng: And a methodological detail tucked in there: all conditions get an eight-thousand-character extract per article, and they verified the gold value lies within it for every scored question. That fixed an earlier artifact where a six-thousand-character cap truncated some long articles and hid the values.

Tom: So from zero percent provenance in the static condition to ninety-nine percent in the end-to-end one. And the gap analysis says version selection was never the hard part once the corpus was versioned.

Lalam: Which raises the bigger question — how far does this generalize beyond French tax law? Page nine steps back and addresses exactly that.

Page 9 of the paper: Tom: We've seen the full arc of the results. Page nine steps back — that's where the conclusion, the limitations, and the impact statement live.

Jane: The conclusion restates the core claim: legal QA should be reframed as a temporally-indexed retrieval problem. Then the limitations section, and it's precise about what the benchmark does and doesn't cover.

Tom: The scope is French tax law, but the authors argue the failure is structural. Any civil-law corpus amended in place and versioned over time exhibits the same three properties. And they point to concrete parallels — the Swiss Fedlex platform, Germany's Gesetze-im-Internet, Belgium's Justel, Luxembourg's Légilux — all exposing versioned statutes with validity dates.

Lu: The method layer is jurisdiction-agnostic; only the data layer is French. And the concurrent German study finding the same phenomenon is independent evidence. They name a Swiss Fedlex replication as the next step.

Meng: On scale, they're candid. Two hundred nine scored questions is small next to LegalBench's hundred sixty-two tasks or KARLBench's two thousand-plus. But they defend the size as a deliberate trade — synthetic or LLM-authored scaling would dilute exactly what makes this benchmark useful: concentrated, expert-reviewed, all-model-hard difficulty.

Tom: And there's a precise scope statement on the failure modes. The single-anchor questions in R3 only exercise current-law substitution. The other three taxonomy modes — future-law leakage, wrong-amendment resolution, multi-version confusion — are left to a future version, though the corpus already contains the not-yet-in-force versions needed to build them.

Jane: There's another boundary in the scope: these tracks target rule application, not legal interpretation. R4 is just a first step toward interpretive benchmarks.

Lalam: The impact statement is restrained, which I appreciate. It frames temporal misgrounding as a reliability failure in high-stakes settings like tax compliance and legal research. The corpus uses only public legislation and case law, no personal data. And they end with the standard caveat: outputs should be verified by a qualified professional, not treated as legal advice.

Lu: That's the right note. This is a measurement paper — it names a failure, quantifies it, and shows the fix — but it doesn't overclaim that the fix makes legal eye trustworthy on its own.

Tom: Alright. Let's wrap this one up.

Conclusion: Tom: So let's close the loop. We've gone through every section now. This paper gave us a name for something that was happening silently in legal eye — temporal misgrounding — and then it went and measured it properly.

Jane: The numbers that will stick with me: three percent from parametric knowledge, two point seven percent with a static corpus, ninety-eight point three once the retrieval knows about dates. And the zero — the static system never once found the applicable version.

Tom: The broader implication is that deployed legal RAG systems, the ones indexing only the current law, aren't just missing edge cases. They're systematically wrong about the past while sounding completely confident. For tax compliance and legal research, that's a serious reliability problem.

Jane: And the benchmark itself is a real contribution — the two hundred nine scored questions, the versioned corpus, the deterministic nugget scoring, all released publicly with model responses and pipeline code. Other researchers can build on the measurement.

Tom: The methodology lesson might be just as important as the finding. By refusing to let an LLM judge the answers, they closed the loop on the very bias they were studying. That's a template for evaluating systems where the model's own priors are part of the problem.

Jane: And it points the field in a concrete direction. Version indexing and date-conditioned retrieval aren't optional extras. They're core architecture for any legal question answering system.

Tom: So that's this paper. It made us rethink how we talk about retrieval in law — the date in the question is part of the query, not decoration.

Jane: Good conversation. Let's move on to the next one.

Tom: Yes, next paper's waiting.