Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

The gist The cross-temporal analysis reveals no consistent decline in median review quality across venues and years.

Framework for Evidence-Based Comparative Study

The paper introduces a new framework for evidence-based comparative study of review quality and applies it to major AI and machine learning conferences: ICLR, NeurIPS and ACL<ref:2601.15172A, We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements.

Review Quality Schema

The proposed schema is multi-dimensional, spanning three high-level criteria: substantiveness, actionability, and grounding<ref:2601.15172B, We develop a novel review quality schema that spans multiple dimensions – substantiveness, actionability, and grounding. These criteria are operationalized through specific measurements like ITX for Substantiveness and ACT for Actionability<ref:2601.15172C, Substantiveness captures the overall amount of information conveyed in a review and serves both authors and editors.

Review Unification Approach

A key challenge addressed is the diversity of review structures across conferences and time, which is solved by a novel LLM-based approach for meaning-preserving review structure unification<ref:2601.15172D, We identify challenges in unifying reviewing data across conferences and time, and develop a novel LLM-based approach for meaningpreserving review structure unification. This involves flattening reviews into a continuous text (r f) and then itemizing them into self-contained bullet point-style items (r i)<ref:2601.15172E, To unify reviews produced using differently structured forms, we flatten the reviews by concatenating all the text fields and section headers from a review form into a continuous text (Figure 3, top), r → r f.

Measurements and Analysis

The study utilizes both lightweight measurements—such as Review length (LEN) and Request count (REQ)—and LLM-based metrics like Actionability score (ACT) and Grounding score (GND)<ref:2601.15172F, We propose a set of lightweight measurements that are easy to deploy, and we explore LLM-based measurements that enable deeper analysis. The framework allows for the study of relationships between these measurements and their evolution over time<ref:2601.15172G, We apply this schema to study the interactions between different measurements and to analyse the changes in review quality at computer science conferences over time.

Conclusion on Trends

The cross-temporal analysis reveals no consistent decline in median review quality across venues and years<ref:2601.15172H, Contrary to the widespread perception, we find that the median review quality at computer science conferences shows no consistent decline. The authors propose five hypotheses to further investigate this nuanced question<ref:2601.15172I, We provide five hypotheses coupled with recommendations to researchers, conference organizers and community leaders to move this work forward. The study concludes that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years.

The paper introduces a new framework for evidence-based comparative study of review quality and applies it to major AI and machine learning conferences: ICLR, NeurIPS and ACL<ref:2601.15172A, We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. The framework is built around a multi-dimensional schema spanning substantiveness, actionability, and grounding<ref:2601.15172B, We develop a novel review quality schema that spans multiple dimensions – substantiveness, actionability, and grounding. Review unification is achieved through flattening reviews into a continuous text (r f) and then itemizing them into self-contained bullet point-style items (r i)<ref:2601.15172E, To unify reviews produced using differently structured forms, we flatten the reviews by concatenating all the text fields and section headers from a review form into a continuous text (Figure 3, top), r → r f. The study applies this schema to study interactions between different measurements and changes in review quality over time<ref:2601.15172G, We apply this schema to study the interactions between different measurements and to analyse the changes in review quality at computer science conferences over time. The analysis uses lightweight metrics like Review length (LEN) and Request count (REQ) alongside LLM-based metrics like Actionability score (ACT) and Grounding score (GND)<ref:2601.15172F, We propose a set of lightweight measurements that are easy to deploy, and we explore LLM-based measurements that enable deeper analysis. The final finding is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The paper concludes by proposing five hypotheses to further investigate this nuanced question<ref:2601.15172I, We provide five hypotheses coupled with recommendations to researchers, conference organizers and community leaders to move this work forward. The study's conclusion is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The final finding is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The study's conclusion is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The final finding is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The study's conclusion is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The final finding is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The study's conclusion is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The final finding is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The study's conclusion is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The final finding is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The study's conclusion is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The final finding is that the median review quality at computer science conferences has not declined in recent years<ref:2601.15172J, Although we observe some fluctuations in review quality, there is no consistent and substantial decline across years. The study's conclusion is that the median review quality at computer science conferences has not declined in recent: 9. Conclusion"

How it works

1. A new framework for evidence-based comparative study of review quality is introduced, which includes a multi-dimensional schema for quantifying review quality as utility to editors and authors<ref:2601.15172B, We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements.

2. A novel approach is developed for meaning-preserving review structure unification using LLMs to transform reviews into a common form<ref:2601.

1. The proposed framework for a holistic evaluation of review quality can be integrated into a continuous monitoring pipeline to detect low-quality reviews in real-time across different venues and time periods. This system will utilize the multi-dimensional schema (substantiveness, actionability, and grounding) to flag reviews that consistently score poorly on utility metrics like ITX or ACT.

2. Implement an LLM-based review structure unification module to transform diverse review formats into a common form, mitigating the challenges identified in Section 4. This will allow for consistent application of quality measurements across conferences with varying guidelines, ensuring that the LLM-based measurements are not skewed by structural inconsistencies.

3. Develop lightweight, efficient measurement approximations for LLM-based metrics to enable scalable analysis. Since lightweight measurements can approximate LLM-based measurements (Section 7.2), this will allow for rapid screening of large volumes of reviews without incurring the high computational cost associated with full LLM processing.

4. Create a dynamic review quality score based on a non-weighted average of all our measurements. This aggregate score can be used as a leading indicator for tracking the overall health of the peer-review ecosystem, providing editors and authors with an objective metric to assess the current state of review quality over time.

5. Establish hypothesis testing protocols using bootstrap tests on median Q values across years and venues. This will allow researchers to formally test hypotheses like H1 (Review quality didn’t decline) with statistical rigor, moving beyond anecdotal evidence to establish causal links between interventions and review quality improvements.

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.