When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning

summary

Video file (mp4)

The gist

The gist The LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred, which suggests that reinforcement-learning-shaped

In short

Researchers tested if large language models can correctly determine which version of a law applies to a case based on when events happened. Findings show models strongly prefer newer laws, regardless of facts, and that reinforcement learning reduces reasoning diversity. Success requires task-specific guidance rather than just better general reasoning.

Key concepts

Temporal Applicable-Law Determination (TALD)
This is the task where an AI must figure out which specific version of a statute applies to a legal case. It involves looking at the facts of the case and matching them against different versions of a law that have changed over time, requiring temporal reasoning.
Policy-Entropy Collapse
This occurs when reinforcement learning training makes an AI's decision-making process too predictable and narrow. Instead of exploring many different ways to solve a problem, the model converges on one specific path—in this case, defaulting to applying the most recent law—reducing its ability to consider alternatives.
Temporal-Law Hints
These are explicit instructions given to an LLM that guide it toward correct temporal reasoning. When models receive strong hints about timing and retroactivity, they show increased exploration of different legal versions, suggesting guidance helps them overcome their default bias.

Terminology used across episodes

This episode discusses

The paper

When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning · Read on arXiv

Institute of Science Tokyo University of Osaka Institute of Science and Technology Center of Juris-Informatics ROIS-DS

Legal reasoning requires identifying which version of a statute applies to a case, typically based on when the legally relevant events occurred rather than when the case is adjudicated. Yet whether large language models (LLMs) can reliably perform such temporal legal reasoning remains largely unexplored. We formalize this task as Temporal Applicable-Law Determination (TALD), and construct a TALD-Bench, a benchmark of 51,956 Chinese civil judgments spanning the enactment of the Civil Code. Evaluating nine advanced LLMs, we identify three key findings. First, models exhibit a strong bias toward newer law, achieving over 70% accuracy on new-law cases but below 25% on old-law cases. Second, models demonstrate substantial knowledge of statutory versions and temporal-effect rules, yet struggle to apply this knowledge to cases. Stronger general reasoning performance does not consistently translate into better TALD accuracy, whereas temporal-law hints yield substantial improvements, suggesting a bottleneck in how models deploy their existing capabilities. Third, temporal-law hints also increase uncertainty during statutory-version reasoning, suggesting that they may help models reconsider their default preference for newer law. Our findings highlight the importance of adapting how LLMs apply existing capabilities, rather than relying solely on improvements in legal knowledge or general reasoning.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning".

Jane: The gist The LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred,

Tom: First, who's behind it and why it matters.

Paper summary: Jane: So we’re wrapping up our look at "When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning." The authors are looking at a huge set of fifty-one thousand nine hundred fifty-six Chinese civil judgments and testing nine different large language models to see why they mess up when it comes to time periods in the law > Page one >

Tom: That’s right. The paper explains that this isn't just about not knowing old laws; it’s about how the reinforcement learning training makes the AI policy collapse, meaning it just defaults to what seems most current instead of checking the case timing > Page two > <ref:2608.14610#pg1>

Lu: The core idea is that TALD—that’s Temporal Applicable-Law Determination—requires models to figure out which version of a statute applies based on when the relevant events actually happened, not just when you ask the question > Page two > <ref:2608.14610#pg1>

Meng: From a deployment standpoint, this convergence on newer statutes is a big problem because real life involves cases from years ago. If we ship that today, it won't work for historical legal questions > Page three > <ref:2608.14610#pg1>

Meng: If we can’t solve this policy misalignment, then whatever accuracy gains we see in their general problem-solving abilities won't matter for complex legal tasks > Page three > <ref:2608.14610#pg1>

Tom: So to wrap up, the authors show that the issue isn't a lack of knowledge about prior versions; it’s a policy collapse caused by reinforcement learning training that makes the AI default to newer statutes regardless of case facts.

Conclusion: Tom: So, we're wrapping up our look at "When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning." The paper investigates why large language models have a strong bias toward newer statutes, even when the facts of a case happened much earlier.

Jane: That’s right. The authors looked at fifty-one thousand nine hundred fifty-six Chinese civil judgments to test nine different models and found that the problem isn't just ignorance of old laws; it's how reinforcement learning training makes the AI policy collapse into applying the most recent law, regardless of when things actually happened.

Lu: The task they define is Temporal Applicable-Law Determination, or TALD. It requires these models to figure out which version of a statute applies based on case facts and the date of those events.

Meng: From a deployment standpoint, that convergence on newer statutes is a big problem because real life involves cases from years ago. If we ship that today, it won't work for historical legal questions.

Lalam: I’m seeing the data show that even when we give the models specific hints about retroactivity or timing—like a strong temporal-law hint—the uncertainty actually goes up during reasoning, which suggests those hints might force them to reconsider their default bias toward newer law.

Tom: That’s interesting. So, the paper isn't just saying "make the model smarter" but rather we have to change how we guide its existing capabilities, not just feed it more general legal knowledge.

Jane: Exactly. The implication for us is that when we build these systems, it’s less about boosting their general reasoning ability and more about aligning their specific policies with the temporal rules of the law itself.

Lu: It points toward a need to engineer task-specific reasoning policies so they don't get stuck in that deterministic loop where they only apply the current statutes.

Meng: If we can’t solve this policy misalignment, then whatever accuracy gains we see in their general problem-solving abilities won't matter for complex legal tasks.

Lalam: I’m confident that by focusing on those version-related reasoning paths instead of just boosting raw reasoning, we can actually get better results on these temporal determination tasks.

Tom: Next time, we’re going to look at how the authors suggest tweaking those hints—the weak versus strong temporal-law hints—and see what that does to the model's behavior.

More episodes

← Home