Low-Rank Ternary Adaptation for Fine-Tuning Transformers

arXiv:2608.24469 · cs.CV, cs.LG · Submitted 2026-08-25 · Read on arXiv

cs.CV, cs.LG

Submitted: 2026-08-25

Updated: 2026-08-25

Comments: Accepted at ECCV 2026. To be published in Volume 17015 of the Lecture Notes in Computer Science series

Code: https://github.com/alexmanoo/ternary_adaptation

License: http://creativecommons.org/licenses/by/4.0/

The gist: Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights.

Terminology

Abstract

Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary adaptation.

Sources

Related papers