RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
cs.AI, cs.CL, cs.LG
Submitted: 2026-10-08
Updated: 2026-10-08
Code: https://github.com/ilyalasy/routerinterp
Project page: https://gabgoh.github.io/ThoughtVectors
Terminology
Sources
- On the Complexity of Neural Computation in Superposition
- gpt-oss-120b & gpt-oss-20b Model Card
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Do Sparse Autoencoders Capture Concept Manifolds?
- The Local Interaction Basis: Identifying Computationally-Relevant and Sparsely Interacting Features in Neural Networks
- A Review of Sparse Expert Models in Deep Learning
- Clusterability in Neural Networks
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- Adaptive Computation Time for Recurrent Neural Networks
- Mixtral of Experts
- Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit
- Scaling Laws for Fine-Grained Mixture of Experts
- DeepSeek-V3 Technical Report
- k-Sparse Autoencoders
- Understanding polysemanticity in neural networks through coding theory
- Efficient Estimation of Word Representations in Vector Space
- Sparse Autoencoders for Hypothesis Generation
- Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
- AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
- Qwen3 Technical Report
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection