When Compression Helps and When It Hurts: Condition-Aware Analysis of Chain-of-Thought Distillation

arXiv:2606.21704 · cs.CL · Submitted 2026-06-19 · Read on arXiv

cs.CL

Submitted: 2026-06-19

Updated: 2026-09-01

Terminology

• Pruning — 62× in this paper · explained in 4 episodes
• Accuracy — 41× in this paper · explained in 1 episode
• Trace — 34× in this paper · explained in 2 episodes
• Benchmark — 14× in this paper · explained in 2 episodes
• Redundancy — 13× in this paper · explained in 2 episodes
• Core — 12× in this paper · explained in 2 episodes
• Distillation — 10× in this paper · explained in 2 episodes
• Fidelity — 9× in this paper · explained in 2 episodes
• Fine-Tuning — 7× in this paper · explained in 5 episodes
• Verification — 7× in this paper · explained in 2 episodes
• Framework Replacement — 6× in this paper
• Semantic Fusion — 6× in this paper
• Reasoning Traces — 5× in this paper · explained in 3 episodes
• Robustness — 5× in this paper · explained in 5 episodes
• Structural Preservation — 5× in this paper
• Structural Constraints — 4× in this paper · explained in 2 episodes
• Structural Editing — 4× in this paper
• Abstraction — 3× in this paper · explained in 1 episode
• Chain-of-Thought (CoT) — 3× in this paper · explained in 4 episodes
• Computation and Language — 3× in this paper
• Ablation — 2× in this paper · explained in 1 episode
• Aggregation — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Commentary — 2× in this paper · explained in 2 episodes
• Perplexity — 2× in this paper · explained in 3 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Calibration — 1× in this paper · explained in 7 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Digital Twin — 1× in this paper · explained in 4 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• LoRA Adapters — 1× in this paper · explained in 3 episodes
• Pairwise Comparisons — 1× in this paper · explained in 2 episodes
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Scaffolding — 1× in this paper · explained in 3 episodes
• Structural Integrity — 1× in this paper · explained in 3 episodes
• Verifier — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
• What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis
• Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
• Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
• Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning
• Training Verifiers to Solve Math Word Problems
• O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
• Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models
• EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensation
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models
• C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness
• Break the Chain: Large Language Models Can be Shortcut Reasoners
• Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy
• The Llama 3 Herd of Models
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Token-Budget-Aware LLM Reasoning
• A*-Thought: Efficient Reasoning via Bidirectional Compression for Low-Resource Settings

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering