What Limits Us? Analyzing Self-Reported Limitations in NLP Research

arXiv:2609.15191 · cs.CL · Submitted 2026-09-14 · Read on arXiv

cs.CL

Submitted: 2026-09-14

Updated: 2026-09-14

Comments: EMNLP 2026 Findings

Code: https://github.com/Sundione/nlp-self-reported-limitations

License: http://creativecommons.org/licenses/by/4.0/

The gist: Since late 2022, a Limitations section has become mandatory at many top-tier NLP conferences.

Terminology

Abstract

Since late 2022, a Limitations section has become mandatory at many top-tier NLP conferences. The growing number of accepted papers at these venues has resulted in a vast corpus of self-reported limitations that cannot all be manually reviewed, yet remains systematically unanalyzed. Therefore, in this paper, we conduct a large-scale analysis of the Limitations sections from ACL and EMNLP papers published between 2020 and 2025 to understand what researchers disclose about their own work. To do so, we implement a novel human-AI framework for iterative hybrid qualitative coding. This framework enables us to investigate trends in self-reported limitations over time, their correlations with specific paper attributes, and the writing patterns that recur around these disclosures. Our findings offer a critical reflection on the diverse reported challenges as well as the self-reporting practices of researchers in the NLP community.

Sources

Related papers