pi squared: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models

arXiv:2604.05114 · cs.CL, cs.AI, cs.LG · Submitted 2026-04-06 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "pi squared: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models".

Jane: The study introduces a pipeline called π2 that curates high-quality, structure-originated reasoning data from Wikipedia tables to significantly improve the long-context reasoning ability of large language models.

Tom: First, who's behind it and why it matters.

Paper summary: Jane: To break down what this paper is actually doing, the pi squared pipeline constructs high-quality reasoning data through three main stages: first, it collects and expands tables from Wikipedia using the MediaWiki Action API, filtering them by size constraints to keep things manageable <ref:2604.05114#pg0,constructs high-quality reasoning data through>.

Lu: And then they generate realistic and multi-hop analytical reasoning questions paired with an executable SQL query for each table, which they then verify using dual-path code execution to ensure accuracy.

Tom: That verification step is crucial; it means the answers aren't just guesses but are checked against an independent Python-implemented solution, which really boosts the quality of the training material.

Meng: I see how that systematic approach addresses data quality issues; getting a reliable source for both the question and the expected answer through code validation seems like a solid engineering foundation.

Lalam: And finally, they backtranslate step-by-step structured reasoning traces as solutions to these QA pairs, which provides the model with those detailed, verifiable reasoning steps given realistic web-search context.

Jane: So in short, the thesis of "pi squared: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models" is that this pipeline creates structured data by extracting and expanding Wikipedia tables, generating verified multi-hop questions with SQL queries, and then backtranslating the structured reasoning traces as solutions for training.

Tom: And what makes it matter is the reported results: when supervised fine-tuning models like GPT-OSS-20B and QWEN3 point 5-35B on this pi squared data, they saw consistent improvements across four long-context reasoning benchmarks and their own pi squared-Bench.

Lu: The paper specifically reports average absolute accuracy gains of plus four point three percent for GPT-OSS-20B and plus two point seven percent for QWEN3-4B on these benchmarks, which shows a measurable uplift in their analytical capabilities <ref:2604.05114#pg0,average absolute accuracy gains of>.

Meng: Those specific numbers give us a concrete measure of effectiveness; it’s not just theoretical improvement, it’s something they've quantified against established long-context reasoning tests.

Lalam: What's also really exciting is that the dataset generated by pi squared facilitates effective self-distillation, meaning smaller models can actually benefit from learning their own generated reasoning traces <ref:2604.05114#pg0>.

Conclusion: Tom: So, to wrap up this discussion on "pi squared: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models," the paper essentially introduces a method for creating high-quality long-context reasoning data by systematically curating structured information from Wikipedia tables.

Jane: The authors claim that this approach directly improves the analytical skills of large language models when they are given extended context, moving past the limitations where simply increasing context size doesn't automatically lead to better analysis.

Lu: The implication here is that providing models with evidence organized in a structured, multi-hop format gives them a much stronger foundation for complex reasoning tasks than unstructured text alone.

Meng: For practical application, this suggests that if we want models to tackle nuanced data analysis tasks, building datasets from structured sources like Wikipedia tables could be a very effective way to guide their development.

Lalam: I think the impact is significant because it shows a path toward training AI that can perform deep structural reasoning rather than just pattern matching based on vast amounts of surface text.

Tom: Exactly, and the authors highlight that even a small set of one hundred high-quality samples already yields meaningful improvements, which validates how efficient this curation pipeline is in producing valuable data.

Jane: So, overall, the paper proposes a concrete way to generate supervision for long-context reasoning tasks using structured evidence derived from real-world data sources like Wikipedia tables.

Virginia Tech

cs.CL, cs.AI, cs.LG

Submitted: 2026-04-06

Updated: 2026-10-02

Code: https://github.com/vt-pi-squared/pi-squared

Importance score: 82/100

The gist: The study introduces a pipeline called π2 that curates high-quality, structure-originated reasoning data from Wikipedia tables to significantly improve the long-context reasoning ability of large

Key concepts

$\pi^2$ Pipeline
A three-stage system designed to build high-quality reasoning data. It sources Wikipedia tables, generates multi-hop analytical questions with SQL queries validated by code, and then creates step-by-step reasoning traces for LLMs.
Synthetic Table Expansion
A technique used in the data curation process where the pipeline attempts to add up to three contextually relevant columns to existing Wikipedia tables if they are too small or too large, ensuring diverse input structures for reasoning tasks.
Dual-Path Code Execution Validation
A rigorous verification step where an LLM generates a query (like SQL) and a Python script independently solves the problem. The answer is only accepted if both execution paths yield the same result, ensuring high accuracy in question generation.

Terminology

Summary

The study introduces a pipeline called π2 that curates high-quality, structure-originated reasoning data from Wikipedia tables to significantly improve the long-context reasoning ability of large language models. This method addresses the limitation where simply extending context capacity does not automatically yield better analytical skills, by synthesizing challenging multi-hop reasoning scenarios grounded in structured evidence.

The gist

A pipeline is presented that constructs high-quality reasoning data through rigorous QA curation: 1) extracting and expanding tables from Wikipedia, 2) generating realistic and multi-hop analytical reasoning questions whose answers are automatically determined and verified through dual-path code execution, and 3) backtranslating step-by-step structured reasoning traces as solutions of QA pairs given realistic web-search context.

Data Curation Pipeline

The π2 pipeline consists of three key stages designed to create realistic long-context reasoning data:

  1. Table collection and possible expansion: This involves sourcing tables from Wikipedia using the MediaWiki Action API, filtering for size constraints (e.g., discarding tables with fewer than 5 rows or more than 30 rows), and implementing a strategy of synthetic table expansion by attempting to add up to three new, contextually relevant columns based on external knowledge when necessary.

  2. QA generation: For each resulting table, the process involves prompting an LLM to generate a realistic and multi-hop analytical reasoning question paired with an executable SQL query. The generated question is judged on complexity, self-containedness, and naturalness using a Likert-5 scale (requiring at least a rating of 3 for complexity). The answer is then validated through dual execution: checking the SQL query result against an independent Python-implemented solution.

  3. Reasoning trace generation: For each validated QA pair, the pipeline gathers evidence from a corpus resembling real-world searching and compacts it to a maximum of 96K tokens for context management. Finally, an LLM is prompted to articulate the structured analytical reasoning steps required to solve the question, given only the compacted natural language context and the verified answer, ensuring that the original structured table is strictly withheld from the context during this phase.

Model Training and Evaluation

The curated dataset facilitates supervised fine-tuning (SFT) of large language models. The authors report consistent improvements across four long-context reasoning benchmarks: LongSeal, LongBench-v2, Oolong, and OfficeQA. Specifically, finetuned models like π2-20B-A3B (from GPT-OSS-20B) and π2-4B (from QWEN3.5-35B) gain average absolute accuracy gains of +4.3% and +2.7% respectively compared to their base models on these benchmarks, with the finetuned GPT-OSS-20B achieving an average improvement of +4.3% and QWEN3-4B achieving +2.70%.

Efficiency and Self-Distillation

The study also investigates the efficiency of the curation pipeline and the potential for self-distillation. Ablation studies show that even a small set of 100 high-quality samples yields meaningful improvements (+1.34% to +2.85%), validating the efficiency of our curation pipeline. Furthermore, the core QA data enables effective self-distillation, where smaller models can benefit from their own generated traces; for instance, GPT-OSS-20B improves its average performance by +4.4% with its own reasoning traces.

Qualitative Analysis and Findings

Qualitative analysis reveals that the finetuned model π2-20B-A3B demonstrates systematic year-by-year verification, showing evidence aggregation across the entire context, thus not being tricked or confused by the question like the base model, which is a core capability targeted by their data curation pipeline. The results confirm that the reasoning patterns learned from π2 align well with deeper computational reasoning during inference when using high reasoning effort configurations. The work concludes that their pipeline produces strong and transferable supervision for structured, multi-hop reasoning tasks.

Conclusion

The authors conclude by stating that they have developed a high-quality long-context reasoning data curation pipeline and synthesized a dataset with both training data and a benchmark π2-Bench. They emphasize the usefulness of their dataset, noting that our code, data, and models are open-source at https://github.com/vt-pi-squared/pi-squared, to facilitate reproducible research. Future work plans include scaling up the dataset and exploring more advanced training strategies.

Ethics Statement

The authors acknowledge ethical considerations regarding data source privacy (using publicly available Wikipedia content) and biases in synthetic data generation, noting that "synthetic data may inadvertently propagate biases present in the underlying model or source content.

Improvements for AI systems

Here are specific improvements that can be made to current AI systems by implementing the methodologies described in the paper, along with what these improved systems would be capable of doing:

  1. Improving Long-Context Reasoning Fidelity:

  2. Enhancing Multi-Hop Analytical Problem Solving:

  3. Enabling Grounded, Step-by-Step Trace Generation:

  4. Facilitating Efficient Knowledge Distillation and Model Transfer:

Specific Improvements and Capabilities of the Enhanced AI System (π2):

These improvements enable the following specific capabilities for the enhanced AI system (π2):

  1. Improved Long-Context Reasoning Fidelity: The system will exhibit significantly higher accuracy when models are tasked with aggregating and synthesizing information across extremely long contexts (up to 128K tokens). It will reduce hallucinations and logical errors that typically plague LLMs when processing massive amounts of text, ensuring the model maintains factual consistency even when information is sparsely distributed.

  2. Enhancing Multi-Hop Analytical Problem Solving: The system will be able to solve complex, multi-step reasoning tasks that require linking disparate pieces of evidence across multiple sources or tables (e.g., SQL data and natural language context). It can perform deep analytical tasks such as comparative analysis, calculating ratios from structured data within a long text, or identifying subtle relationships between entities embedded in a large document.

  3. Enabling Grounded, Step-by-Step Trace Generation: The system will generate transparent reasoning traces that mimic expert human workflows (e.g., Step 1: Understand the question, Step 2: Search for condition X, Step 3: Verify result Y). This allows users to audit the model's thought process, identify exactly where it struggled (e.g., failed in information retrieval vs. logical deduction), and debug complex failures in real-time.

  4. Facilitating Efficient Knowledge Distillation and Model Transfer: The system introduces a mechanism where large teacher models generate high-quality reasoning traces, which are then used to supervise the fine-tuning of smaller student models (e.g., GPT-OSS-20B). This allows smaller, more efficient models to inherit the advanced reasoning patterns learned from larger systems, leading to substantial performance gains without requiring massive computational resources for every model iteration.

Sources

Related papers