Magnitude Profile Pruning: Calibration-Free Structured Attention Head Removal for Transformer Compression

arXiv:2609.26177 · cs.CL, cs.AI, cs.LG · Submitted 2026-08-11 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-08-11

Updated: 2026-08-11

Comments: Accepted as a regular paper at IEEE ICMLA 2026; to appear in the conference proceedings

Code: https://github.com/Kasun-Dewage/Magnitude-Profil

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

Sparsity — 50× in this paper · explained in 4 episodes
Pruning — 42× in this paper · explained in 4 episodes
Calibration — 35× in this paper · explained in 7 episodes
Perplexity — 29× in this paper · explained in 3 episodes
Transformer — 16× in this paper · explained in 2 episodes
Accuracy — 8× in this paper · explained in 1 episode
Magnitude Profile — 8× in this paper
Fine-Tuning — 3× in this paper · explained in 5 episodes
Grouped Query Attention — 3× in this paper
Robustness — 3× in this paper · explained in 5 episodes
Adaptation — 2× in this paper · explained in 1 episode
Regularization — 2× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
For Multi-Head Attention — 1× in this paper
Transformer Models — 1× in this paper · explained in 2 episodes

Sources

Related papers