LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillation

The gist: LINGUDISTILL proposes an adapter-free distillation method that restores native linguistic capability in Vision-Language Models by leveraging a frozen LM teacher through layer-wise KV-cache sharing and selective distillation, recovering about 10% of the performance lost on language and knowledge benchmarks while maintaining comparable performance on vision-heavy tasks.

Introduction

Adapting pretrained language models (LMs) into vision-language models (VLMs) can degrade their native linguistic capability due to representation shift and cross-modal interference introduced during multimodal adaptation, such as representation shift and cross-modal interference introduced during multimodal adaptation. Prior recovery approaches typically introduce additional modules that act as intermediate alignment layers to maintain or isolate modality-specific subspaces, which increases architectural complexity, adds parameters at inference time, and limits flexibility across models and settings. LINGUDISTILL is a cross-modal knowledge distillation method to restore language capability in VLMs. A key challenge is that the teacher only operates on text, so its outputs are not aligned with vision-conditioned generation. To address this, we introduce a KV-cache sharing mechanism between the teacher and student decoders. After training, the teacher is removed, resulting in a standard VLM with improved language capability and no additional parameters or inference cost.

KV Sharing Architecture

The KV-cache sharing mechanism proceeds as follows:

1. The student decoder Φ processes the full multimodal sequence X and produces KV caches: The student decoder Φ processes the full multimodal sequence X and produces KV caches.

2. The teacher decoder omega reprocesses the same text prompt to form its query states: The teacher decoder omega reprocesses the same text prompt to form its query states.

3. The teacher omega directly reuses the student KV cache at every layer: The teacher omega directly reuses the student KV cache at every layer.

4. The teacher attends using its computed queries over the transported student memory: The teacher attends using its computed queries over the transported student memory.

This design allows the teacher to attend to the student’s multimodal context via shared KV caches, effectively conditioning it on multimodal input.

Selective Distillation Objective

The student (Φ) is optimized with a typical KD mixture of frozen LM’s (omega) soft distillation and its own hard next-token supervision. The final objective is L = 1 / omegapos ∑ (b,t)∈omegapos α(db) L(b,t) soft + (1 − α(db)) L(b,t) hard. This objective allows the teacher’s supervision signal to be sourced more from languagerelevant data, while relying more on hard supervision for visual-heavy sources.

Experimental Setup and Results

The experiments use the instruction-tuned nanoVLM-460M-8k model as the VLM (student) and its instruction-tuned LM backbone, SmolLM-360M-Instruct (teacher). The training uses a selective distillation objective described in Sec. 3.2. Selective KD (Ours) resolves the trade-off by routing the distillation signal based on data type. Table 7 reports benchmark scores across all variants. Selective KD achieves the strongest gains coupled with a controlled perception ability degradation. For instance, on language and knowledge-heavy benchmarks, ScienceQA reaches 0.676 (+14.2%).

Discussion

LINGUDISTILL recovers performance on language and knowledge benchmarks by ∼10%, while exhibiting comparable degradation in vision-heavy tasks relative to standard fine-tuning methods. The key insight is that distillation helps whenever a task needs knowledge, regardless of whether it also involves reading text. For OCRBench, the model corrects nonsense strings into real words when using LINGUDISTILL because The teacher’s language signal makes the model better at understanding text but worse at copying exactly what it sees.

Conclusion

LINGUDISTILL proposes an adapter-free distillation framework that restores native linguistic capability by leveraging a frozen LM teacher through layer-wise KV-cache sharing and selective distillation. Our results demonstrate that LINGUDISTILL recovers performance on language and knowledge benchmarks while maintaining comparable performance on vision-heavy tasks, surpassing standard finetuning objective with zero additional parameters. These findings suggest that targeted cross-modal distillation provides a simple and effective approach to preserving backbone capabilities in multimodal systems without introducing additional modules.

A Training Configuration

The shared training configuration across all variants includes:

(a) Hard loss across KD-strength variants.

(b) Soft loss across KD-strength variants.

Table 4 lists hyperparameters shared across all variants. All experiments use nanoVLM-full (Wiedmann et al., 2025) with a total size of 460M parameters, which pairs a SigLIP2-B/16 vision encoder (Tschannen et al., 2025) with a SmolLM2-360M-Instruct language decoder (Allal et al., 2025). The teacher is a frozen copy of the original SmolLM2-360M-Instruct backbone used in nanoVLM-full.

D Sequence Length Analysis

We train with a maximum sequence length of 1024 tokens and a single image per example. The overall mean across all sources is under 120 tokens, meaning the 1024-token limit provides ample headroom for most training examples.

E Detailed OCR Analysis

The regression in Key Information Extraction and Doc-oriented VQA together make up 89 of the 126-point total drop (71%), even though they are only 40% of the benchmark. We see the same pattern in DocVQA, where the biggest losses are in table/list questions (−109), layout questions (−84), and free-text extraction (−89).

F Expanded & Ablation Results

Selective Distillation achieves the best language gains with contained vision degradation. We attribute LINGUDISTILL’s success to selective distillation. The main LINGUDISTILL setting reaches the lowest CE curve overall, while High KD stays slightly above it and Low KD remains the highest throughout training <ref:2604.008291,9Figure 2Training-loss analysis for the three selective distillation variants. Left: the CE term is lowest for the main LINGUDISTILL setting, with High KD slightly above it and Low KD clearly worse <ref:2604.

1. To recover native language capability in VLMs without additional modules, implement LINGUDISTILL to utilize the original frozen LM as a teacher. This results in a linguistically improved VLM with zero additional modules and recovers ∼10% of the performance lost on language and knowledge benchmarks.

2. Implement layer-wise KV-cache sharing between the student decoder (Φ) and the frozen teacher decoder (omega) to enable vision-conditioned supervision. This mechanism allows the teacher to access the same multimodal context as the VLM and produce vision-aware supervision signals without modifying architecture or adding parameters at inference time.

3. Employ data-dependent weighting in the selective distillation objective, using data-dependent weighting, where distillation is primarily applied to language-intensive data. This ensures that LINGUDISTILL restores linguistic capability by applying distillation only on languagecapability recovering data while preserving the original structure of the VLM on visual-heavy or general multimodal data.

4. Utilize the selective KD objective based on source categorization, where Language-heavy (8) receive the full KD signal and OCR/doc-heavy (9) receive CE only, as summarized in Table 1. This strategy allows for a superior trade-off: it maximizes language gains while preserving visual capability significantly better than uniform distillation.

5. Apply distillation hyperparameter tuning based on Figure 2, where the optimal setting is found by balancing the trade-off: The main LINGUDISTILL setting reaches the lowest CE curve overall, while High KD stays slightly above it and Low KD remains the highest throughout training.

Turning a pretrained language model (LM) into a vision-language model (VLM) through multimodal fine-tuning often erodes its native language ability, a form of catastrophic forgetting that shows up even on text-only tasks. This loss is hard to undo with further fine-tuning, and existing remedies add adapters or alignment modules that increase architectural complexity and inference cost. We propose LinguDistill, an adapter-free knowledge distillation method that uses the original frozen LM as the teacher during multimodal post-training. To let a text-only teacher supervise vision-conditioned outputs, we introduce layer-wise KV-cache sharing, which exposes the teacher to the student's multimodal representations without changing either architecture. We then apply distillation selectively, on language-heavy data only, so the teacher restores linguistic ability while the student keeps its visual grounding on document and OCR tasks. LinguDistill recovers the language and knowledge performance lost during multimodal fine-tuning, matching the original VLM on average over text-only benchmarks (ARC, HellaSwag) and exceeding it on ScienceQA, while keeping vision-heavy performance close to standard fine-tuning. Since the teacher is dropped after training, the final model adds no parameters and no inference cost. More broadly, our results show that a model's own pre-adaptation backbone is a practical teacher for undoing forgetting, suggesting a simple recipe for keeping language ability intact as models are extended to new modalities.