Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance

arXiv:2511.13254 · cs.CL · Submitted 2025-11-17 · Read on arXiv

cs.CL

Submitted: 2025-11-17

Updated: 2026-09-18

Code: https://github.com/facebookresearch/llm_souping

License: http://creativecommons.org/licenses/by/4.0/

The gist: Large Language Models (LLMs) have displayed remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive computational resources and

Terminology

Abstract

Large Language Models (LLMs) have displayed remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive computational resources and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. We observe that previous souping approaches can lead to collapse in precision-sensitive LLM capabilities. In this paper, we introduce SoCE, a principled approach for model souping to overcome this shortcoming. The proposed method utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous approaches, our method leverages the observation that different clusters (or categories) of points within a benchmark often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that SoCE improves performance and robustness across multiple domains and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.

Sources

Related papers