Tree-of-Concerns: Hierarchical Multi-Agent Debate for Unstated-Limitation Extraction in Scientific Critique

arXiv:2608.20777 · cs.CL · Submitted 2026-08-21 · Read on arXiv

cs.CL

Submitted: 2026-08-21

Updated: 2026-09-21

Comments: Accepted in the Findings of EMNLP 2026

License: http://creativecommons.org/licenses/by/4.0/

The gist: As scientific literature grows and papers increasingly under-report limitations, multi-agent LLMs offer a promising approach to systematically uncover these hidden failure modes.

Terminology

Abstract

As scientific literature grows and papers increasingly under-report limitations, multi-agent LLMs offer a promising approach to systematically uncover these hidden failure modes. Here, we introduce Tree-of-Concerns, a multi-agent framework that deploys specialized skeptic personas, each operating through a category-specific analytical lens, as parallel debate trees to extract unstated limitations from scientific papers. Each persona conducts structured, evidence-grounded argumentation, while a Panel Review mechanism re-evaluates each surviving claim from all five perspectives to correct category drift and severity miscalibration. Through retrieval-free, single-paper experiments on ToC-Bench, our benchmark of 414 research papers with 1,905 unstated limitations, sourced from reviewer-reported weaknesses and follow-up citation critiques, we demonstrate that ToC improves precision by 79% and coverage by 11% relative to the strongest baseline, surfacing specific, evidence-grounded concerns that support reviewers in systematic evaluation.

Sources

Related papers