Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts
cs.CL, cs.LG
Submitted: 2026-10-08
Updated: 2026-10-08
Terminology
Sources
- Program Synthesis with Large Language Models
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
- DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
- Measuring Massive Multitask Language Understanding
- Training Compute-Optimal Large Language Models
- Harder Tasks Need More Experts: Dynamic Routing in MoE Models
- Mixtral of Experts
- Scaling Laws for Fine-Grained Mixture of Experts
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- BASE Layers: Simplifying Training of Large, Sparse Models
- Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
- Post-Trained MoE Can Skip Half Experts via Self-Distillation
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering