Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs

Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements, and this paper presents a framework for quantizing VLMs for efficient inference on resource-constrained hardware. The gist is that a novel 2.7-bit-per-parameter format, S3D8, combined with a quantization-aware training pipeline based on the model's own generations allows for compressing models like Llama 3.2 11B Vision Instruct to 3.7 GB while preserving strong performance on visual question answering tasks.

The Core Problem and Solution

The primary challenge addressed is achieving sub-3-bit weight compression with efficient Arm CPU execution without access to the original training data, a difficult constraint in the multimodal setting where aggressive quantization often causes severe model degradation. To solve this, the framework introduces two central components: first, a Synthetic Data Generation pipeline that uses the model itself to create training samples for knowledge distillation; and second, a novel numerical format called S3D8 designed for efficient Arm CPU inference. This approach aims to achieve effective sub-3-bit quantization with limited model degradation.

The Quantization-Aware Training (QAT) Pipeline

The QAT procedure is constructed to train a quantized student model to match the behavior of the original full-precision teacher model on synthetic data. The process involves:

1. Constructing a synthetic multimodal training set using images from ImageNet and generating corresponding textual responses by querying the teacher model.

2. Applying a consistent QAT procedure after specifying the target numerical format to distill the original model using this synthetic data, minimizing the per-sample distillation loss defined as: LKD(I, x, y) = 1/y Σ t DKL (p(t)T p(t)S).

3. The prompt sampling strategy is carefully designed to ensure diversity and generalization by randomly applying instruction templates with probability ptemp (0.75), using generic image-comprehension questions, and biasing responses toward short and medium lengths via length probes.

The S3D8 Numerical Format

S3D8 is a novel 2.7-bit-per-parameter format that packs three signed weights into each byte through a shared 5-bit centroid index and decodes them to INT8 for inference. The format is defined by the equation: W̃iⱼ = αi ⋅ sⱼ ⋅ Cq⌊i/3⌋,j, i mod 3, where α is per-channel scale, s is the sign indicator bit, C are per-matrix centroids (32×3), and q are quantization indices. To improve decoding efficiency on Arm CPUs, S3D8 employs three design choices: packing separate output channels into each byte; exploiting the 64-entry table lookup support in Arm CPUs to look up signed values directly; and reordering sign indicator bits to require just five logical instructions to construct the three lookup indices.

Performance and Runtime Analysis

The paper validates the S3D8 format across various quantization procedures (direct casting, GPTQ, QAT) against scalar baselines like INT and Lloyd-Max. Results show that S3D8 outperforms the standard INT with block-scaling, achieving approximately 22% extra compression for the same task performance. Furthermore, runtime performance benchmarking on Arm CPUs demonstrates practical efficiency: S3D8 dequantization takes 133μs (16.5 GB/s) for a vision MLP up-projection on the Pixel 8a using 5 cores, which is lower than an INT8 copy's 310μs (21.2 GB/s).

Downstream Task Evaluation

The effectiveness of S3D8 is measured across standard visual question answering tasks (VQAv2, ChartQA, DocVQA, AI2D). Table 1 shows that at a similar model size, S3D8 achieves favorable downstream performance against INT and other formats. Specifically, the overall format at 2.68 bits per parameter leads to an average task degradation of 0.083 compared to the original bfloat16 model. The results confirm that QAT applied to S3D8 further increases average task performance to 0.661, and runtime analysis supports its viability for mobile deployment on devices like the Android Pixel 8a and Graviton4 CPU.

Ablation Studies

The study includes several ablation experiments to isolate the impact of different design choices. For instance, keeping selected layers in higher precision (INT8) after QAT was tested, showing that while it reduces the training KL loss, it does not meaningfully improve downstream performance. The analysis also compares different GPTQ parameterizations (absmax vs.

Based on the provided paper, here are the specific improvements that can be made to AI systems and what those improved systems can achieve:

1. Improved Inference Efficiency on Resource-Constrained Hardware: The introduction of the S3D8 2.7-bit-per-parameter format enables significantly reduced memory footprint (e.g., compressing Llama 3.2 11B Vision Instruct to 3.7 GB) and faster inference speeds on Arm CPUs compared to standard INT8 or GPTQ baselines, especially for vision prefill and text generation where it shows a speedup over INT8 (up to 25% on Pixel 8a).

2. Reduced Quantization Error with Aggressive Compression: The paper demonstrates that the QAT pipeline, specifically when utilizing the S3D8 format, achieves better performance-compression trade-offs than direct casting or standard GPTQ. This allows for achieving sub-3-bit weight compression (2.7 bits/parameter) while maintaining strong visual question answering (VQA) performance on standard benchmarks.

3. Data-Free Quantization Training Pipeline: The framework introduces a novel synthetic multimodal data generation pipeline that allows for quantization-aware training without access to the original model's training data or recipe. This means models can be aggressively quantized for mobile deployment using only the pre-trained model itself as a teacher, bypassing the need for expensive downstream fine-tuning datasets.

4. Enhanced Generalization via Synthetic Prompting: The synthetic data generation includes a sophisticated prompting strategy that randomly samples from both instruction-tuned templates and generic image-comprehension questions (with optional instruction blocks). This forces the quantized model to learn diverse response styles and instruction adherence, leading to better downstream generalization compared to models trained with fixed prompts.

5. Hardware-Aware Numerical Format Co-Design: The S3D8 format is specifically designed for Arm CPU execution by optimizing the bit packing and lookup table decoding process. It utilizes SIMD instructions (like TBL) to fuse sign indicators and centroid lookups into just 5 logical instructions, minimizing runtime overhead and maximizing memory bandwidth utilization during dequantization.

6. Versatile Deployment Across Mobile Architectures: The implementation provides a full VLM inference pipeline for both Android (Pixel 8a) and server-grade Arm CPUs (Graviton4). This ensures that the optimized model can be deployed effectively across various hardware tiers, with performance metrics showing practical speedups for single-user text generation on mobile devices.

7. Improved Multimodal Reasoning Fidelity: Empirical results show that S3D8 retains strong performance across a set of standard VQA tasks (VQAv2, ChartQA, DocVQA) and even demonstrates successful recovery of correct final answers in complex multiple-choice scenarios (AI2D), indicating that the aggressive quantization does not critically damage the model's ability to perform complex visual reasoning.

In summary, these improvements result in AI systems that are significantly more efficient for deployment on edge devices while maintaining high accuracy. The improved AI systems can perform:

1. Real-time, low-latency multimodal inference on mobile phones without requiring massive memory or significant battery drain.

2. High-quality visual question answering and document reading tasks directly on the device with minimal performance degradation compared to full precision models.

3. Efficient deployment of large Vision-Language Models (VLMs) in environments where cloud access is limited or unavailable, such as in remote industrial settings or personal mobile devices.