Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging

arXiv:2609.32303 · cs.AI, cs.LG · Submitted 2026-09-26 · Read on arXiv

cs.AI, cs.LG

Submitted: 2026-09-26

Updated: 2026-09-26

Code: https://github.com/EliSpectre/UI-MOPD

Terminology

• Distillation — 26× in this paper · explained in 2 episodes
• Benchmark — 21× in this paper · explained in 2 episodes
• Accuracy — 13× in this paper · explained in 1 episode
• Grounding — 11× in this paper · explained in 3 episodes
• On-Policy Distillation — 10× in this paper · explained in 3 episodes
• Alignment — 8× in this paper · explained in 1 episode
• Agentic Reasoning — 5× in this paper · explained in 1 episode
• GRPO — 5× in this paper · explained in 6 episodes
• Model Merging — 5× in this paper · explained in 2 episodes
• Reinforcement Learning — 4× in this paper · explained in 4 episodes
• Self-distillation — 2× in this paper · explained in 6 episodes
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Agentic Task — 1× in this paper · explained in 1 episode
• Agentic Tasks — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Core — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes

Sources

• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
• Reward-Gated On-Policy Distillation
• On Predictability of Reinforcement Learning Dynamics for Large Language Models
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Rethinking On-Policy Distillation of Large Language Models II: One Training Example
• When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
• Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
• Editing Models with Task Arithmetic
• MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
• ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
• Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
• UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents
• Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
• The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
• Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
• MiMo-V2-Flash Technical Report
• TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents
• MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
• OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection