Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities

The gist: OOLONG introduces a benchmark for long-context reasoning tasks that require analyzing individual chunks of text at an atomic level and aggregating these analyses to answer distributional questions, which is crucial as model context lengths continue to grow.

OOLONG Benchmark Structure

OOLONG is separated into two task sets: OOLONG-synth, which consists of naturalistic synthetic tasks where components can be easily ablated, and OOLONG-real, a downstream setting requiring reasoning over real-world conversational data<ref:2511.02817#pg4>. OOLONG requires models to perform reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations<ref:2511.02817#pg4>. The benchmark demands that models reason over large quantities of text through a multi-step problem where they must identify parts of the input relevant to the question, perform a subtask at each relevant part (e.g. a simple classification) and then pool the results from all the subtasks to generate the final answer<ref:2511.02817#pg5>.

OOLONG-synth Design

OOLONG-synth is constructed by constructing challenging corpus-level questions over existing in-context learning (ICL) datasets<ref:2511.02817#pg4>. The data collection involves 10 common text classification datasets, and these are split into validation and test tasks<ref:2511.02817#pg4>. Context window construction involves sampling a distribution over label classes so the model cannot use any information about the true distribution over labels, as well as sampling dates and user IDs<ref:2511.02817#pg5>. Questions are constructed in three types: counting questions concern simple statistical properties of the label distribution, user information questions require additional cross-reference with the user ID field, and timeline questions ask about changes in distribution before or after a date, between years, or between months across years<ref:2511.02817#pg5>.

OOLONG-real Data

OOLONG-real poses challenging information aggregation questions over real data, specifically transcripts from Dungeons & Dragons role-playing game shows<ref:2511.02817#pg4>. These transcripts involve several levels of conversation, from out-of-character chitchat to rules discussion to incharacter actions and speech, and they can be unscripted, involving tangents or side channels<ref:2511.02817#pg4>. The dataset is compiled from the Critical Role Dungeons and Dragons Dataset (CRD3), using episode transcripts with player names labeled<ref:2511.02817#pg6>. Context windows for OOLONG-real range from one to 24 episode transcripts, covering input lengths of 55K to 1.3M tokens<ref:2511.02817#pg7>.

Evaluation and Results

The evaluation involves a baseline, parsing answers using specified output formats, and scoring based on exact match for certain answer types or a score of score(ˆy) = 0.75y−yˆ for numerical answers<ref:2511.02817#pg5>. Frontier models struggle with the task, as GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K<ref:2511.02817#pg4>. The study found that identification and aggregation of information is the bottleneck, not labeling when ablating settings in OOLONG-synth<ref:2511.02817#pg5>. Furthermore, temporal questions are consistently the most challenging for models on OOLONG-synth<ref:2511.02817#pg5>. The results show a significant drop in performance at higher context windows across both splits<ref:2511.02817#pg7>.

Model Behavior Analysis

The analysis of model behavior reveals that for OOLONG-synth, the apparent strategy of labeling each example before deciding which are relevant results in running out of context tokens<ref:2511.02817#pg5>. In contrast, traces for OOLONG-real do not show these pathologies<ref:2511.02817#pg7>. DeepSeek R1 shows a discrepancy, performing below the random baseline on OOLONG-synth because the model’s apparent strategy of labeling each example before deciding which are relevant results in running out of context tokens<ref:2511.02817#pg7>. The paper concludes that OOLONG is a usefully challenging evaluation of long-context reasoning abilities<ref:2511.02817#pg5>.

Future Directions and Limitations

The authors argue that posing questions over a large block of context represents a realistic use scenario<ref:2511.02817#pg5>. They also note that the performance differences between these models is not due to differing ability to perform the classification task when providing labels in-context<ref:2511.02817#pg10>. The work suggests that there is still a long way to go in designing robust long-context aggregation capabilities for LLMs<ref:2511.02817#pg7>. The study's findings are supported by results from prior work on related benchmarks, such as RULER and HELMET<ref:2511.02817#pg10>. The use of D&D data is noted as a novel approach, as we are the first to use fan annotations of gold labels and to consider the generation of these statistics as a task in its own right<ref:2511.02817#pg10>. The work acknowledges that some of these incomplete traces do contain an answer in a small number of cases<ref:2511.02817#pg10>.

Conclusion

OOLONG introduces a challenging long-context information aggregation benchmark in two parts<ref:2511.02817#pg7>. OOLONGsynth uses synthetic aggregation tasks over ICL data to enable finer-grained control of the benchmark settings, while OOLONG-real poses questions over real long-context conversational data and human-annotated labels<ref:2511.02817#pg7>. On both splits, models struggle, with performance dropping with increasing context length even when controlling for the potential compounding of mislabeling errors<ref:2511.02817#pg7>. We see substantial headroom between strong open weights models and API-based models on this task, particularly OOLONG-synth<ref:2511.02817#pg7>.

1. Improved long-context reasoning capability through multi-step information aggregation: The OOLONG benchmark requires models to reason over large quantities of examples, to perform both classification and counting in-context, and to reason over temporal and user relations, enabling models to perform multi-hop reasoning over long inputs in a single pass.

2. Enhanced real-world conversational data reasoning: By including OOLONG-real, the system can reason over real world conversational data, specifically "transcripts from live-action Dungeons & Dragons shows," allowing it to handle complex, unscripted context where information is not easily decomposable.

3. Improved robustness against context length limitations: The research demonstrates that even when controlling for mislabeling errors, models struggle with aggregation as input grows, suggesting the system can be trained or prompted to employ strategies that mitigate token exhaustion, such as identifying and aggregation of information is the bottleneck.