3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs

arXiv:2603.01376 · cs.LG, stat.ML · Submitted 2026-03-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs".

Jane: The gist The 3BASiL framework introduces an efficient one-shot post-training method for Sparse plus Low-Rank (S + LR) decomposition of Large Language Models (LLMs) that addresses performance degradation seen in…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We've seen how 3BASiL addresses the performance degradation in Sparse plus Low-Rank decomposition of LLMs, and now we're going to look at the core idea of this paper '3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs' <ref:2603.01376#pg1,3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs>.

Jane: The thesis is that they introduce 3BASiL, which is a novel three-Block Alternating Direction Method of Multipliers method designed to minimize the layer-wise reconstruction error with convergence guarantees <ref:2603.01376#pg1,to minimize the layer-wise reconstruction error with convergence guarantees>.

Lu: They formulate the problem by minimizing the Frobenius norm error between original and decomposed weights subject to sparsity and rank constraints, breaking it down into three variable sets: sparse component, low-rank component, and original weights.

Meng: The iterative ADMM framework is used to optimize those three sets, meaning they derive updates for the sparse component S(t+one), the low-rank component L(t+one), and the constrained copy D(t+one) through closed-form solutions based on the augmented Lagrangian function <ref:2603.01376#pg3>.

Lalam: The computational complexity per iteration is noted as O(N three), which gives us a concrete idea of how demanding this iterative process is for large models <ref:2603.01376#pg1>.

Tom: After setting up the ADMM, they introduce the transformer-matching procedure called TM, which is described as a novel memory-efficient refinement procedure that jointly optimizes sparse and low-rank components across transformer layers.

Jane: This TM step refines the components by aligning transformer block outputs with the dense model, essentially acting as an intermediate loss function between layer-wise proxies and the true end-to-end loss function.

Lu: This joint optimization across layers is key because it significantly improves sparse component quality with minimal computational cost by directly leveraging transformer-level outputs, addressing a major limitation in current Sparse plus Low-Rank methods.

Meng: And they stress that this TM procedure is universally applicable and can enhance any existing Sparse plus Low-Rank decomposition method.

Lalam: This allows for joint refinement of both sparse and low-rank components at the transformer level, which provides a more effective initialization for downstream adaptation.

Conclusion: Tom: So to wrap up this discussion on 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs, we've covered the main points about how this method tackles performance degradation <ref:2603.01376#pg1,3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs>.

Jane: The authors are Mehdi Makni, Xiang Meng, and Rahul Mazumder from Operations Research Center Massachusetts Institute of Technology.

Lu: The core contribution is the introduction of 3BASiL-TM, an efficient one-shot post-training method for Sparse plus Low-Rank decomposition of LLMs that addresses the performance gap <ref:2603.01376#pg1,3BASiL-TM, an efficient one-shot post-training method for>.

Meng: It means we have a framework that uses a novel ADMM combined with a transformer matching refinement step to get high quality results with convergence guarantees.

Lalam: The paper shows that this approach is reproducible and provides detailed implementation specifications, which is always important for new methods in the field.

Tom: Overall, 3BASiL-TM offers substantial speedups in compression runtime compared to prior methods and it’s validated across various benchmarks on models like Llama8B <ref:2603.01376#pg1>.

Jane: It suggests one route for optimal compression involves unfolding the LLM compression into those three minimization steps: layer-wise reconstruction, transformer-matching, and then LoRA fine-tuning.

Lu: The implication is that you can achieve high quality Sparse plus Low-Rank decomposition for LLMs in a single post-training step using this framework.

Meng: It means we have a method that uses an iterative ADMM combined with a transformer matching refinement step to get high quality results with convergence guarantees.

Lalam: This shows how to take a complex weight decomposition problem and solve it algorithmically, which gives us confidence in the final compressed weights we get.

Massachusetts Institute of Technology

cs.LG, stat.ML

Submitted: 2026-03-02

Updated: 2026-10-08

Code: https://github.com/mazumder-lab/3BASiL

Importance score: 90/100

The gist: The gist The 3BASiL framework introduces an efficient one-shot post-training method for Sparse plus Low-Rank (S + LR) decomposition of Large Language Models (LLMs) that addresses performance

Key concepts

Sparse plus Low-Rank (S + LR) Decomposition
This is the goal: breaking down the massive weight matrices of LLMs into two parts—a sparse part (few non-zero values) and a low-rank part (highly compressible structure). This decomposition makes the model much smaller and faster to run while retaining most of its performance.
3-Block Alternating Direction Method of Multipliers (ADMM)
This is the core optimization engine. ADMM is an iterative mathematical technique used here to find the best sparse and low-rank components by minimizing errors. It works by breaking down a complex problem into smaller, manageable parts that are solved iteratively until a stable solution is found.
Transformer Matching (TM)
This is a refinement step that aligns the decomposed model with the original dense model across transformer layers. It acts as an intermediate loss function, jointly optimizing both the sparse and low-rank components at the transformer level to ensure better overall compression quality.

Terminology

Summary

The gist The 3BASiL framework introduces an efficient one-shot post-training method for Sparse plus Low-Rank (S + LR) decomposition of Large Language Models (LLMs) that addresses performance degradation seen in existing methods by combining a novel 3-Block Alternating Direction Method of Multipliers (ADMM) with a transformer-matching refinement step.

How it works

The core of the proposed method is 3BASiL, which is a novel 3-Block Alternating Direction Method of Multipliers (ADMM) method designed to minimize the layer-wise reconstruction error with convergence guarantees The problem formulation involves minimizing the Frobenius norm error between original and decomposed weights subject to sparsity and rank constraints. This is achieved by decomposing the weight decomposition problem into three variable sets—sparse component, low-rank component, and original weights—optimized within an iterative ADMM framework. The updates for the sparse component S(t+1), low-rank component L(t+1), and sparse component’s constrained copy D(t+1) are derived through closed-form solutions based on the augmented Lagrangian function. The computational complexity is noted as O(N 3) per iteration.

Transformer matching and Universality

A significant addition to the framework is the transformer-matching (TM) procedure, which is described as a novel (memory-efficient) refinement procedure that jointly optimizes sparse and low-rank components across transformer layers. This step refines the components by aligning transformer block outputs with the dense model, acting as an intermediate loss function between layer-wise proxies and the true end-to-end loss function. Crucially, this TM procedure is universally applicable and can enhance any existing (S + LR) decomposition method. This allows for joint refinement of both sparse and low-rank components at the transformer level, which provides a more effective initialization for downstream adaptation.

Empirical Validation and State-of-the-Art Results

Numerical experiments demonstrate that 3BASiL-TM is a new state-of-the-art method for one-shot (S + LR) decomposition of LLMs. Specifically, the method reduces WikiText2 perplexity gap to dense model by over 30% compared to prior methods for a Llama8B model under a (2:4 Sparse + 64 LR) configuration. Furthermore, the method achieves over 2.5x faster compression runtime on an A100 GPU compared to SOTA (S + LR) method. The results show that 3BASiL-TM significantly improves LLM evaluation benchmarks including perplexity of different datasets and various zero-shot tasks.

Theoretical Guarantees

The paper establishes a novel convergence guarantee for the 3-Block ADMM approach. Theorem 1 ensures that the decomposition converges as long as the penalty parameter ρt increases sufficiently rapidly. The proof in Appendix A provides a rigorous derivation showing that both sequences of matrices S(t) and L(t) are Cauchy sequences, leading to convergence to a matrix W¯ = S¯ + L¯ where S(t) + L(t) → W¯ as t → ∞.

Limitations

The paper discusses the limitation of its work at the end of Section 6. Specifically, it notes that it remains to explore dedicated methods that can algorithmically allocate different sparsity/rank configurations to different layers to further improve efficiency-utility computations tradeoffs. While the method is versatile and theoretically sound, dedicated methods for algorithmic allocation of configurations are still needed.

The 3BASiL framework presents a highly-efficient (S + LR) decomposition algorithm with theoretical convergence guarantees and provides high-quality solutions to the layer-wise decomposition problem. It further refines these decomposed weights with its novel transformer matching step TM that can enhance any (S + LR) decomposition. This shows that one route for optimal compression results is to unfold the LLM compression into three minimization steps: layer-wise reconstruction, transformer-matching, and LoRA fine-tuning and. The method's performance is validated across various configurations and models on benchmarks like WikiText2 perplexity and zero-shot tasks. This approach offers substantial speedups in compression runtime compared to prior methods and. The framework is reproducible with detailed implementation specifications provided in Appendix B and the code is available online. The work adheres to the NeurIPS Code of Ethics.

--- Page 1 ---

The gist The 3BASiL framework introduces an efficient one-shot post-training method for Sparse plus Low-Rank (S + LR) decomposition of Large Language Models (LLMs) that addresses performance degradation seen in existing methods by combining a novel 3-Block Alternating Direction Method of Multipliers (ADMM) with a transformer-matching refinement step.

How it works

The core of the proposed method is 3BASiL, which is a novel 3-Block Alternating Direction Method of Multipliers (ADMM) method designed to minimize the layer-wise reconstruction error with convergence guarantees. The problem formulation involves minimizing the Frobenius norm error between original and decomposed weights subject to sparsity and rank constraints. This is achieved by decomposing the weight decomposition problem into three variable sets—sparse component, low-rank component, and original weights—optimized within an iterative ADMM framework. The updates for the sparse component S(t+1), low-rank component L(t+1), and sparse component’s constrained copy D(t+1) are derived through closed-form solutions based on the augmented Lagrangian function. The computational complexity is noted as O(N 3) per iteration.

Empirical Validation and State-of-the-Art Results

Numerical experiments demonstrate that 3BASiL-TM is a new state-of-the-art method for one-shot (S + LR) decomposition of LLMs.

Improvements for AI systems

  1. textbf3BASiL-TM for Joint Optimization of Sparse and Low-Rank Components with Enhanced Quality and Speed Stability. The proposed transformer matching (TM) procedure is described as an intermediate loss function between layer-wise proxies and the true end-to-end loss function, which creates a more accurate proxy of the original loss function by directly minimizing the discrepancy between the original and compressed transformer outputs, resulting in higher performance pruned models. This allows for joint refinement of both sparse and low-rank components across transformer layers.

  2. textbf3BASiL Convergence Guarantees for Robust Layer-wise Reconstruction. The paper introduces a novel convergence guarantee that ensures the decomposition converges as long as we choose penalty parameter ρt that increases sufficiently rapidly, providing theoretical assurance for the iterative 3-Block ADMM approach when optimizing (S + LR) decomposition.

  3. textbfEfficient One-Shot Compression and Fast Runtime across Diverse Configurations. The method achieves significant computational advantages, showing over 2.5x faster compression runtime on an A100 GPU compared to SOTA (S + LR) method and demonstrating superior trade-offs, such as achieving the best compression-performance trade-off under (3:8 + LR) configurations among different (S + LR) methods.

  4. textbfSmart Initialization for Downstream Adaptation via LoRA. The low-rank components obtained from compression serve as smart initialization for subsequent LoRA fine-tuning, which allows the model to recover lost performance during task adaptation by using these components as an effective starting point for LoRA.

Sources

Related papers