Not All 4-bit Quantizers Are Equal: Deployment-Time Mitigation of PII Leakage in Fine-Tuned Small Language Models
cs.CR, cs.AI, cs.LG
Submitted: 2026-08-02
Updated: 2026-08-02
Comments: 10 pages and 5 tables. The source code and per-seed experimental results are publicly available at https://github.com/CristhianKapelinski/quantizer-pii-mitigation. Accepted for publication at SBSeg 2026
Code: https://github.com/CristhianKapelinski/quantizer-pii-mitigation
License: http://creativecommons.org/licenses/by/4.0/
The gist: Organizations fine-tune small language models on private data and then compress them to 4 bits for resource-efficient deployment.
Terminology
Abstract
Organizations fine-tune small language models on private data and then compress them to 4 bits for resource-efficient deployment. We show that the compression method also affects privacy. What separates the methods is not the bit width but whether they tune their rounding on a small sample of text, the calibration corpus. On our primary model, when each planted record's own opening text is used as the prompt, the two calibration-based methods we test, Activation-aware Weight Quantization (AWQ) and Gradient-based Post-Training Quantization (GPTQ), each reproduce none of the planted records, while the calibration-corpus-free GGUF Q4 K M format reproduces 5.3% of them. Tracked across five open models with 0.5-7 billion parameters, AWQ leaks least at every size and in both families, with little accuracy loss at 3-7 billion. Controlled experiments associate the difference with calibration-induced rounding error in channels involved in rare-token prediction. Choosing the 4-bit method is therefore a deployment-time privacy decision, not only a question of speed and quality.
Sources
- Quantization-Robust LLM Unlearning via Low-Rank Adaptation
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- How Quantization Impacts Privacy Risk on LLMs for Code?
- Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct
- QUAIL: Quantization Aware Unlearning for Mitigating Misinformation in LLMs
- Scalable Extraction of Training Data from (Production) Language Models
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models
- Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA
Related papers
- SoK: AI-Augmented Binary Reversing
- Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
- Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
- Efficient Fuzzy PSI under One-Sided Assumptions
- Sealing the Audit-Runtime Gap for LLM Skills
- Token Composition: A Graph Based on EVM Logs