MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

arXiv:2511.00421 · cs.CL, cs.AI, cs.LG · Submitted 2025-11-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts".

Jane: MEDRECT introduces a novel, cross-lingual benchmark for medical error detection and correction, focusing on Japanese and English clinical texts.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, we're starting with the basics here; the title itself, "MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts," really tells us exactly what this paper is about—it's setting up a test to see how well language models handle medical mistakes across Japanese and English.

Jane: And looking at the authors, Naoto Iwase, Hiroki Okuyama, and Junichiro Iwasawa from Preferred Networks, Inc. and Nagoya University gives us a sense of the academic rigor behind this new testing framework.

Lu: From my perspective at Tsinghua, this cross-lingual approach is incredibly interesting because it moves beyond just testing models in one language; evaluating transfer across Japanese and English contexts is a much richer data point for understanding real-world global medical AI deployment.

Meng: I'm curious about the practical setup here; how are they structuring the test to ensure that both the Japanese and English datasets have a comparable balance of errors versus correct information?

Lalam: The paper introduces this benchmark as a systematic framework, which means it’s not just throwing texts at models randomly; it's building a reproducible structure for evaluating error detection, localization, and correction subtasks.

The paper's summary: Tom: Moving on to the actual content of "MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts," the paper explains that they formulated medical error handling into three clear steps: finding the error, pinpointing where it is in a sentence, and then fixing it.

Jane: That decomposition is really smart because it lets researchers see exactly where a model struggles—is it failing to spot an error at all, or is it struggling with the actual extraction part?

Lu: The paper describes using two main datasets, MEDRECT-ja with six hundred sixty-three texts from the Japanese Medical Licensing Examinations and MEDRECT-en with four hundred fifty-eight texts from the MEDEC MS Subset Test, which helps make their cross-lingual evaluation systematic.

Meng: I see they created a novel scalable methodology for data construction, using two LLMs to synthesize questions into candidate samples and then employing validation models to filter for specific quality ranges. That sounds like a lot of work on the data side.

Lalam: They used this automated pipeline to create both the Japanese and English datasets, ensuring that they maintain a similar error-to-no-error ratio of approximately fifty-five:forty-five across both languages.

The paper's improvements: Tom: Now for the parts where they show what makes their approach better; the authors highlight that reasoning models perform substantially better than standard architectures, showing up to a thirteen point five percent relative improvement in error detection and a fifty-one point zero percent improvement in sentence extraction compared to non-reasoning models.

Jane: That comparison between the reasoning and non-reasoning groups is really telling; it suggests that simply having more parameters isn't enough, but having the right kind of reasoning capability is what really helps these systems improve their accuracy on complex tasks.

Lu: The paper also points out a performance gap when evaluating cross-lingual capabilities, specifically noting five to ten percent performance differences between English and Japanese, although this gap narrowed for the models that possessed reasoning skills.

Meng: They also tested targeted LoRA fine-tuning and found asymmetric improvements in error correction, seeing a gain of +zero point zero seven eight for Japanese and +zero point one six eight for English in those specific tasks while keeping the core reasoning abilities intact.

Lalam: One of the most compelling improvements discussed is that by using their novel reasoning synthesis training data with LoRA, they can substantially boost bilingual error correction performance, creating a clear path toward safer AI systems that can actually show their work.

Conclusion: Tom: So, wrapping up the discussion on "MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts," the main point is that this benchmark provides a vital resource for building more accurate and reliable medical LLMs across different languages.

Jane: It really shows us that we need to move beyond simple knowledge recall when building these tools; we need models that can actually perform nuanced error handling and correction within clinical contexts.

Lu: I think the implication here is huge for how we approach global health AI, because if reasoning patterns transfer effectively across languages, it opens up possibilities for equitable medical support worldwide.

Meng: From an engineering viewpoint, the paper's suggested pathway using targeted fine-tuning with LoRA gives us a concrete strategy to improve performance without needing massive retraining efforts on the entire base model.

Lalam: Ultimately, this benchmark sets a reproducible framework that allows the community to systematically evaluate these capabilities and develop AI systems that are not only accurate but also transparent about their reasoning steps.

Preferred Networks, Inc. · School of Medicine, Nagoya University

cs.CL, cs.AI, cs.LG

Submitted: 2025-11-01

Updated: 2026-09-30

Comments: 16 pages. To appear at the EMNLP 2026 Workshop on Open Reasoning Across Cultures & Languages (ORACLE)

Code: https://github.com/pfnet-research/medrect

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 86/100

The gist: MEDRECT introduces a novel, cross-lingual benchmark for medical error detection and correction, focusing on Japanese and English clinical texts.

Key concepts

Cross-lingual Benchmark
This is a standardized test using medical texts in two different languages (Japanese and English) to evaluate AI models. It moves beyond single-language tests to see if a model's understanding of medical errors can be applied when the language changes, which is vital for global healthcare AI.
Error Localization
This subtask requires an LLM not only to find an error in a clinical sentence but also to precisely extract the exact part of that sentence where the mistake occurs. This tests a model's ability to pinpoint specific problems rather than just identifying that something is wrong.
Inverted Cross-lingual Pattern
This finding means that when fine-tuning a model, it performed better on English tasks than its original Japanese baseline. This suggests that the reasoning skills learned from one language can actually boost performance in another language, indicating fundamental medical understanding is language-independent.

Terminology

Summary

MEDRECT introduces a novel, cross-lingual benchmark for medical error detection and correction, focusing on Japanese and English clinical texts. This research addresses the critical need for reliable Large Language Models (LLMs) in healthcare by evaluating their ability to handle nuanced reasoning failures beyond simple knowledge recall. The benchmark is significant because it provides a reproducible framework that moves beyond monolingual or single-language evaluations, offering crucial insights into how medical reasoning capabilities transfer across different linguistic and cultural contexts, which is essential for developing safer and more equitable global medical AI systems.

Benchmark Structure and Subtasks

MEDRECT formulates medical error handling as three progressive subtasks: Error Detection, Error Localization (sentence extraction), and Error Correction. This decomposition allows for a fine-grained evaluation of model capabilities, where sentence extraction is only applicable to texts containing an error. The benchmark consists of two primary datasets: MEDRECT-ja (663 texts) derived from the Japanese Medical Licensing Examinations (JMLE) and MEDRECT-en (458 texts) curated from the MEDEC MS Subset Test. This cross-lingual setup provides a systematic framework for evaluating models across different languages while maintaining comparable error/no-error balance, with both datasets showing similar error-to-correct ratios of approximately 55:45.

Data Construction Pipeline

The creation of the benchmark utilizes a Novel Scalable Methodology to reduce resource intensity. For MEDRECT-ja, the pipeline systematically transforms 287 clinical case questions from JMLE (2024 and 2025) into candidate samples through four sequential processes:

  1. Data Synthesis: Using two LLMs to automatically transform MCQA into clinical texts, creating CORRECT samples and ERROR samples based on answer choices.

  2. Quality Filtering: Employing 11 validation models to filter for appropriate difficulty, retaining samples where error detection accuracy falls within a specific range (1/11 ≤ accuracy ≤ 7/11) and minimizing the gap between detection and extraction accuracy.

  3. Model Deduplication: Selecting one sample from each synthesis model for duplicate pairs to maintain balanced representation.

  4. Final Quality Screening: Using LLM-as-a-Judge (Gemini 2.5 Pro) to classify samples based on five quality dimensions, excluding any sample scoring '1' (problematic) on any dimension.

Model Evaluation and Key Findings

The evaluation involved 9 contemporary LLMs, categorized into Reasoning Models and Non-reasoning Models. A key finding is that reasoning models substantially outperform standard architectures; specifically, reasoning models showed up to a 13.5% relative improvement in error detection and a 51.0% improvement in sentence extraction compared to non-reasoning counterparts. Furthermore, cross-lingual evaluation revealed 5-10% performance gaps from English to Japanese, though these disparities were smaller for reasoning models. Targeted LoRA fine-tuning yielded asymmetric improvements: +0.078 for Japanese and +0.168 for English in error correction performance, while preserving reasoning capabilities.

Cross-Lingual Transfer and Fine-Tuning Impact

The study highlights significant insights into cross-lingual knowledge transfer. While proprietary models generally perform better on English, the fine-tuned Qwen3-32B + LoRA model demonstrated an inverted cross-lingual pattern, achieving superior English performance (0.718 average score) compared to its Japanese baseline (0.627), representing a 30.5% relative improvement in English error correction. This suggests that medical reasoning patterns learned from Japanese clinical scenarios effectively transfer to enhance English capabilities, indicating that fundamental error detection skills transcend language barriers. The fine-tuned model also surpassed human expert performance on structured tasks, such as sentence extraction and error correction on the original MEDEC benchmark.

Error Type Analysis

Performance by error type revealed distinct difficulty hierarchies. Medication dosage emerges as the most challenging category, with average performance around 70%, whereas Medication selection represents the most tractable category. Crucially, reasoning capabilities showed differential impacts: "The Qwen3-32B think vs. no-think comparison reveals particularly large gaps in History taking sentence extraction (68.1% vs. 36.2%) and Physical findings (68.4% vs. 37.8%), indicating that explicit reasoning processes are especially beneficial for tasks requiring contextual interpretation and clinical observation synthesis. The qualitative analysis further showed that fine-tuning enhanced empathetic patient communication while simultaneously helping models restrain from overcorrecting already-accurate text," addressing practical deployment concerns.

Conclusion

MEDRECT establishes itself as the "first comprehensive cross-lingual benchmark for medical error detection and correction, providing a reproducible framework and resources for developing safer medical LLMs across languages.

Improvements for AI systems

Here are specific improvements for AI systems based on the MEDRECT benchmark and its findings:

  1. Improved Reasoning Capabilities in Medical LLMs: By fine-tuning models (e.g., Qwen3-32B + LoRA) using reasoning synthesis data, AI systems can achieve a substantial relative improvement in error detection (up to 13.5%) and sentence extraction (up to 51.0%).

  2. Enhanced Cross-Lingual Medical Error Correction: The new benchmark allows for the systematic evaluation of cross-lingual transfer. Fine-tuned models can leverage medical reasoning patterns from one language (e.g., Japanese) to significantly boost performance in another language (English), showing a substantial 30.5% relative improvement on English correction scores compared to their base performance in Japanese, despite training data size differences.

  3. Task-Specific Error Localization: AI systems can move beyond simple binary error detection to perform precise sentence extraction (localization) with up to 81.5% accuracy on the MEDRECT-ja dataset for reasoning models, allowing for the pinpointing of exactly which part of a complex clinical narrative contains the mistake.

  4. Robust, Scalable Benchmark Creation: Researchers can now build high-quality, cross-lingual medical error correction benchmarks without relying heavily on resource-intensive manual annotation by medical experts. An automated pipeline synthesizing data from standard exams (like JMLE) and existing datasets (like MEDEC) provides a reproducible framework for creating similar benchmarks across different languages.

  5. Targeted Model Enhancement: The study identifies that explicit reasoning processes are crucial for complex tasks like History taking and Physical findings, where non-reasoning models fail significantly. AI development efforts should prioritize incorporating structured, step-by-step reasoning mechanisms (like the 'think' mode in Qwen3) to improve performance in these contextually demanding areas.

  6. Improved Clinical Safety and Transparency: Fine-tuned systems can be trained to identify false positive errors—where correct text is flagged as erroneous—a critical safety feature for reducing clinician burnout and improving system utility. Furthermore, the fine-tuning process preserves explainable reasoning, allowing clinicians to trace the model's logical steps, increasing trust in medical AI deployment.

  7. Domain-Specific Error Profiling: By analyzing error type distributions (e.g., identifying that Medication dosage is a consistently challenging category), AI systems can be specifically targeted for further development in areas with persistent numerical precision challenges, leading to more reliable dosing recommendations and fewer critical errors in those domains.

Sources

Related papers