Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement
cs.LG, cs.AI, cs.CL
Submitted: 2026-08-26
Updated: 2026-08-26
License: http://creativecommons.org/licenses/by/4.0/
The gist: Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference.
Terminology
Abstract
Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from superposition, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78% improvement over the strongest baseline.
Sources
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling
- Toy Models of Superposition
- GPT-4 Technical Report
- Ministral 3
- The Llama 3 Herd of Models
- No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
- Qwen2.5 Technical Report
- Superpose Task-specific Features for Model Merging
- Improving Dictionary Learning with Gated Sparse Autoencoders
- Polysemanticity and Capacity in Neural Networks
- Instruction-Following Evaluation for Large Language Models
- Gemini: A Family of Highly Capable Multimodal Models
- Model merging with SVD to tie the Knots
- Qwen3 Technical Report
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks