Locality-Aware Redundancy Pruning for LLM Depth Compression
cs.LG, cs.AI
Submitted: 2026-05-27
Updated: 2026-08-30
Comments: EMNLP 2026 Main Accepted
Code: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning
License: http://creativecommons.org/licenses/by/4.0/
The gist: Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency.
Terminology
Abstract
Large language models are known to contain representational redundancy across network depth, making depth pruning an effective approach for improving inference efficiency. Existing one-shot pruning methods rely on local layer importance or fixed redundancy assumptions across architectures. We propose Locality-Aware Redundancy Pruning (LoRP), a training-free one-shot depth pruning framework guided by representation locality. We show that inter-layer redundancy can be either localized or globally distributed depending on the LLM architecture. To characterize this phenomenon, we introduce Representation Locality Score (RLS), derived from global inter-layer hidden-state similarity. Using a small calibration set, LoRP computes pairwise layer similarity, clusters layers by representational similarity, and allocates pruning according to residual intra-cluster redundancy. Experiments across diverse LLM families show improvements in both perplexity and downstream task accuracy. Official github repository: https://github.com/daniel-eai/LoRP-Locality-Aware-Redundancy-Pruning/
Sources
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- The Llama 3 Herd of Models
- Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods
- Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning
- 2 OLMo 2 Furious
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- LLaMA: Open and Efficient Foundation Language Models
- Qwen3 Technical Report
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks