BioDivergence: A Benchmark and Evaluation Framework for Hidden Contextual Contradictions in Biomedical Abstracts

arXiv:2606.11208 · cs.CL, cs.AI · Submitted 2026-04-23 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-04-23

Updated: 2026-08-30

Comments: The authors have decided to withdraw this manuscript because substantial revisions to the study design, analysis, and presentation are needed before the work is suitable for public dissemination. A revised version may be submitted separately after these issues have been fully addressed

Code: https://github.com/eliashossain001/biodivergence

License: http://creativecommons.org/licenses/by/4.0/

The gist: Biomedical findings often seem to conflict across studies, but many of these differences are context-dependent rather than true contradictions.

Terminology

Abstract

Biomedical findings often seem to conflict across studies, but many of these differences are context-dependent rather than true contradictions. Variations in cohort, geography, assay protocol, disease subtype, and clinical setting can make both claims locally valid. Existing NLI and scientific claim-verification benchmarks reduce such cases to entailment, contradiction, or neutral, failing to capture the contextual structure behind divergence. To address this, we introduce BioDivergence, an evaluation framework with a six-class conflict taxonomy, a 13-axis divergence ontology, and four structured outputs per claim pair: conflict type, divergence axes, dominant confounder, and reconciliation explanation. We release BioDivergence-Silver-v1.0, an article-disjoint silver benchmark of 11,865 claim pairs across five biomedical domains, alongside a legacy deduplicated variant for comparison. Results show notable ranking differences between the two variants, with the fine-tuned reference model dropping about 12 points under the article-disjoint setting, while Mistral-7B-Instruct-v0.3 achieves 0.5523 accuracy and 0.3894 contextual-F1 on the 842-example primary test set. BioDivergence offers a more faithful way to distinguish contextual divergence from direct contradiction and to separate article-level memorization from genuine task learning.

Sources

Related papers