HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion

arXiv:2609.32581 · cs.CL, cs.LG · Submitted 2026-09-26 · Read on arXiv

cs.CL, cs.LG

Submitted: 2026-09-26

Updated: 2026-09-26

Terminology

• Benchmark — 10× in this paper · explained in 2 episodes
• Ablation — 7× in this paper · explained in 1 episode
• Regularization — 7× in this paper · explained in 3 episodes
• Transformer — 5× in this paper · explained in 2 episodes
• Calibration — 3× in this paper · explained in 7 episodes
• Cooperation — 3× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Alignment — 2× in this paper · explained in 1 episode
• Generalization — 2× in this paper · explained in 13 episodes
• Mixture of Experts — 2× in this paper · explained in 2 episodes
• Mixture-of-Experts (MoE) — 2× in this paper · explained in 7 episodes
• Parameter Efficiency — 2× in this paper · explained in 3 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Communication Overhead — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Sparsity — 1× in this paper · explained in 4 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
• Evaluating Large Language Models Trained on Code
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Training Verifiers to Solve Math Word Problems
• Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
• FastMoE: A Fast Mixture-of-Expert Training System
• Measuring Massive Multitask Language Understanding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
• Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
• AURA: A Diagnostic Framework for Tracking User Satisfaction of Interactive Planning Agents
• Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• DeepSeek-V3 Technical Report
• Load Balancing Mixture of Experts with Similarity Preserving Routers
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
• Qwen3 Technical Report

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering