The Dynamics of Continuous Mixture Collapse in Language Models

arXiv:2609.02049 · cs.LG, cs.CL · Submitted 2026-09-02 · Read on arXiv

cs.LG, cs.CL

Submitted: 2026-09-02

Updated: 2026-09-02

Terminology

• Benchmark — 9× in this paper · explained in 2 episodes
• Embeddings — 7× in this paper · explained in 2 episodes
• Transformer — 6× in this paper · explained in 2 episodes
• Latent Reasoning — 3× in this paper · explained in 2 episodes
• Reasoning Traces — 3× in this paper · explained in 3 episodes
• Trace — 3× in this paper · explained in 2 episodes
• Transformer Architecture — 3× in this paper · explained in 2 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Self-consistency — 2× in this paper · explained in 2 episodes
• Chain-of-Thought (CoT) — 1× in this paper · explained in 4 episodes
• Chain-of-Thought (CoT) Reasoning — 1× in this paper · explained in 2 episodes
• Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Dynamical regimes — 1× in this paper · explained in 2 episodes
• GRPO — 1× in this paper · explained in 6 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Self-distillation — 1× in this paper · explained in 6 episodes

Sources

• Soft Tokens, Hard Truths
• LLM Latent Reasoning as Chain of Superposition
• SeLaR: Selective Latent Reasoning in Large Language Models
• Gemma 4 Technical Report
• Continuous Chain of Thought Enables Parallel Exploration and Reasoning
• Training Large Language Models to Reason in a Continuous Latent Space
• Large Language Models are Zero-Shot Reasoners
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models
• CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking
• Training Continuous Chain of Thought Models: A Tale of Two Regimes
• Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
• SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization
• Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought
• Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought
• Text Generation Beyond Discrete Token Sampling

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks