Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs

arXiv:2609.32821 · cs.AI, cs.LG · Submitted 2026-09-26 · Read on arXiv

cs.AI, cs.LG

Submitted: 2026-09-26

Updated: 2026-09-26

Code: https://github.com/wyy-code/SRR

Terminology

• Accuracy — 32× in this paper · explained in 1 episode
• Model Merging — 16× in this paper · explained in 2 episodes
• Benchmark — 12× in this paper · explained in 2 episodes
• AUROC — 11× in this paper · explained in 2 episodes
• Consistency — 5× in this paper · explained in 3 episodes
• Selective Router Repair — 5× in this paper
• Trace — 5× in this paper · explained in 2 episodes
• Calibration — 4× in this paper · explained in 7 episodes
• Alignment — 3× in this paper · explained in 1 episode
• Average and Task Arithmetic — 3× in this paper
• Distillation — 3× in this paper · explained in 2 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Cosine Similarity — 2× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Redundancy — 2× in this paper · explained in 2 episodes
• Scaling Laws — 2× in this paper · explained in 4 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Mixture-of-Experts (MoE) — 1× in this paper · explained in 7 episodes
• Mixture-of-Experts (MoE) Models — 1× in this paper · explained in 3 episodes
• On-Policy Distillation — 1× in this paper · explained in 3 episodes
• Quantization — 1× in this paper · explained in 12 episodes
• Reconstruction Error — 1× in this paper · explained in 2 episodes
• Sparsity — 1× in this paper · explained in 4 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectors
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Training Verifiers to Solve Math Word Problems
• FeatCal: Feature Calibration for Post-Merging Models
• Measuring Massive Multitask Language Understanding
• Editing Models with Task Arithmetic
• Mixtral of Experts
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• DeepSeek-V3 Technical Report
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Discovering Physical Directions in Weight Space: Composing Neural PDE Experts
• E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
• Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
• Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation
• Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Qwen3 Technical Report
• Representation Surgery for Multi-Task Model Merging
• ST-MoE: Designing Stable and Transferable Sparse Expert Models

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection