SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
cs.LG, cs.AI
Submitted: 2025-10-10
Updated: 2026-09-07
Comments: paper is accepted at TMLR Journal: https://openreview.net/forum?id=3nZb43fvAQ
Code: https://github.com/mit-han-lab/llm-awq
Project page: https://comeusr.github.io/SQS
License: http://creativecommons.org/licenses/by/4.0/
The gist: Compressing large-scale neural networks is essential for deploying models on resource-constrained devices.
Terminology
Abstract
Compressing large-scale neural networks is essential for deploying models on resource-constrained devices. Most existing methods adopt weight pruning or low-bit quantization individually, often resulting in suboptimal compression rates to preserve acceptable performance drops. We introduce a unified framework for simultaneous pruning and low-bit quantization via Bayesian variational learning, which achieves higher compression rates than prior baselines while maintaining comparable performance. The key idea is to employ a spike-and-slab prior to induce sparsity and model quantized weights using Gaussian Mixture Models (GMMs) to enable low-bit precision. Due to the intractability of the objective involving spike-and-slab priors with GMMs, we derive an efficient approximation that facilitates effective compression with minimal accuracy loss. In theory, we provide a consistent result for our proposed variational approach to a sparse and quantized deep neural network. Extensive experiments on compressing ResNet, BERT-base, Llama3.2, and Qwen2.5 models show that our method achieves higher compression rates than a line of existing methods with comparable performance drops. Project page: https://comeusr.github.io/SQS Webpage.
Sources
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- High-Accuracy Low-Precision Training
- Low-bit Model Quantization for Deep Neural Networks: A Survey
- LLaMA: Open and Efficient Foundation Language Models
- Training and Inference with Integers in Deep Neural Networks
- To prune, or not to prune: exploring the efficacy of pruning for model compression
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks