UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization, and Distillation
cs.LG
Submitted: 2026-02-09
Updated: 2026-08-28
Comments: Accepted as conference paper at EMNLP 2026 Main
Code: https://github.com/jvonrad/UniComp
License: http://creativecommons.org/licenses/by/4.0/
The gist: Model compression is increasingly essential for deploying large language models (LLMs), yet existing comparative studies largely focus on pruning and quantization evaluated primarily on
Terminology
Abstract
Model compression is increasingly essential for deploying large language models (LLMs), yet existing comparative studies largely focus on pruning and quantization evaluated primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation. UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis. Through evaluation of seven compression techniques across over 40 datasets, we observe (i) a consistent knowledge bias, where factual recall is largely preserved while multi-step reasoning, multilingual, and instruction-following capabilities degrade; (ii) a deployment-critical performance-reliability decoupling, where retained performance does not indicate preserved safety, fairness and privacy; and (iii) that task-specific calibration can yield up to 50% relative improvement in reasoning performance in pruned models.
Sources
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Training Verifiers to Solve Math Word Problems
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Distilling the Knowledge in a Neural Network
- LLM Pruning and Distillation in Practice: The Minitron Approach
- Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks