TTPO: Test-Time Policy Optimization

arXiv:2608.27448 · cs.CL · Submitted 2026-08-27 · Read on arXiv

cs.CL

Submitted: 2026-08-27

Updated: 2026-08-27

Code: https://github.com/ZJU-REAL/TTPO

Terminology

• Distillation — 43× in this paper · explained in 2 episodes
• GRPO — 40× in this paper · explained in 6 episodes
• Pseudo-labels — 18× in this paper · explained in 2 episodes
• Thinking mode — 13× in this paper · explained in 2 episodes
• Benchmark — 12× in this paper · explained in 2 episodes
• Reinforcement Learning — 12× in this paper · explained in 4 episodes
• Self-distillation — 12× in this paper · explained in 6 episodes
• Accuracy — 7× in this paper · explained in 1 episode
• Generalization — 6× in this paper · explained in 13 episodes
• On-Policy Distillation — 5× in this paper · explained in 3 episodes
• Ablation — 4× in this paper · explained in 1 episode
• On-Policy Self-Distillation — 4× in this paper
• Adam — 3× in this paper · explained in 2 episodes
• Failure Modes — 3× in this paper · explained in 3 episodes
• Pseudo-labeling — 3× in this paper · explained in 3 episodes
• Test-Time Policy Optimization — 3× in this paper
• Consistency — 2× in this paper · explained in 3 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Credit assignment — 1× in this paper · explained in 3 episodes
• Distribution Shift — 1× in this paper · explained in 5 episodes
• Knowledge Distillation — 1× in this paper · explained in 11 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• LoRA Adapters — 1× in this paper · explained in 3 episodes
• On-policy Self-Distillation (OPSD) — 1× in this paper · explained in 2 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
• Reward Shaping — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
• Test-Time Reinforcement Learning for GUI Grounding via Region Consistency
• Consensus as Privileged Context for Label-Free Self-Distillation
• MiniLLM: On-Policy Distillation of Large Language Models
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
• Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
• On-Policy Self-Distillation without Any Supervision
• Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting
• STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens
• Self-Distilled Policy Gradient
• Decoupled Weight Decay Regularization
• Self-Distilled Agentic Reinforcement Learning
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• OpenAI GPT-5 System Card
• Kimi K3: Open Frontier Intelligence
• Qwen3.5-Omni Technical Report

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering