Specializing Without Forgetting: Analyzing Knowledge Preservation in Multilingual Model Adaptation

Parameter alignment strategies mitigate catastrophic forgetting when specializing multilingual models into language-family experts by systematically comparing five layer-aware methods against unregularized baselines across diverse languages and tasks. This research demonstrates that the optimal parameter alignment strategy is task-specific, with layer freezing best preserving comprehension, post-hoc reversion yielding the strongest translation gains, and L2 regularization consistently maintaining or improving held-out perplexity.

How it works

The study introduces family-expert Continual Pretraining (CPT), a paradigm that organizes training data by language families to enable targeted, distributed expert training. This approach scales language coverage per expert while limiting intra-expert interference, allowing for flexible scaling to new settings and recovering multilingual generalization that unconstrained expert training sacrifices. The core challenge addressed is catastrophic forgetting in this setting, which occurs when experts suffer from cross-lingual forgetting and parameter divergence from the shared initialization.

To mitigate this drift, five layer-aware parameter alignment strategies are instantiated to either regularize parameter updates or correct model weights post-training. These strategies include:

1. Hard layer freezing: This enforces layer boundaries during training by freezing the middle layers while allowing the first and last layers to receive gradient updates, which prevents middle-layer drift, at the cost of reducing the model’s capacity to absorb new language information.

2. Soft regularization (Layer-Range L2): This applies layer-dependent penalty strengths to the learning objective, where a high penalty is set for middle layers (e.g., λmid = 0.05) and low penalties for outer layers, offering a soft alternative to layer freezing during training.

3. Post-hoc weight reversion: This involves resetting the weights of the model’s middle layers back to the base model’s pre-trained weights after training, which is shown to recover general capabilities without requiring any retraining.

4. Model merging (Expert Soup): This strategy merges five vanilla family experts into a single unified model by uniformly averaging their weights, which is shown to produce stronger comprehension retention than any individual family expert checkpoint in some cases.

Evaluation and Findings

The methods are systematically compared against two unregularized CPT baselines: Dense CPT, which trains a single model jointly on all languages without mitigation, and Family Expert (inspired by X-ELM), which trains one expert per family without regularization or post-hoc correction. Evaluation spans five language families (Slavic, Germanic, Indic, Austronesian, Romance) and 32 training languages across four evaluation axes: perplexity (proxy for language acquisition), reading comprehension (Belebele), physical reasoning (Global-PIQA), and translation quality (FLORES-200).

The results show that parameter alignment substantially reduces forgetting over unregularized baselines at minimal cost to language acquisition. Specifically, layer freezing is noted as best preserving downstream performance on average for reading comprehension, while post-hoc reversion yields the strongest translation gains. Layer-Range L2-SP is highlighted as the strategy that consistently maintains or improves held-out perplexity across all five families.

Task-Specific Trade-offs and Causal Analysis

The research maps a nuanced language acquisition–knowledge forgetting tradeoff, indicating that alignment strategy selection should be layer-aware and driven by the target application rather than a single aggregate metric. A causal analysis of layer drift confirms that middle-layer drift is the strongest causal contributor to comprehension degradation, aligning with the design assumptions. However, for translation quality (FLORES), first- and middle-layer interpolation have little effect on ChrF in either translation direction, whereas restoring final-layer drift substantially improves translation quality. This mismatch suggests that layer locations responsible for comprehension forgetting are not necessarily the same locations that control translation behavior.

Generalization to Unseen Languages

The study probes within-family generalization by evaluating held-out languages withheld from training in each family. Dense CPT increases held-out perplexity substantially across every family, while Layer-Range L2-SP and Expert Soup are the only strategies that consistently match or improve upon the base model. Conversely, vanilla family experts degrade held-out perplexity in every family, most sharply for Austronesian languages. The findings indicate that family-level CPT does not yield transferable comprehension gains for held-out relatives.

Deployment Guidelines

The paper derives practical deployment guidelines linking each alignment strategy to the settings it best serves. For example, layer freezing is recommended when comprehension and reasoning are prioritized, while post-hoc reversion is best suited for situations where strong translation quality is required. The optimal constraint type and location remain task-dependent, suggesting that "CPT strategy selection should be driven by the target setting (such as translation-heavy, comprehension-critical, balanced, or broad coverage) rather than by a single aggregate metric.

Here are the specific, actionable improvements for AI systems derived from this research, categorized by strategic application:

---

)1. Implement a Task-Specific Parameter Alignment Framework (The Why):

Instead of relying on a single alignment strategy for all applications, develop an automated system that selects the optimal parameter alignment method based on the intended downstream task.

[Improvement]: Design a meta-controller module that takes a task profile (e.g., high-stakes translation, complex physical reasoning, broad low-resource language coverage) as input and maps it to one of five strategies:

• Use Layer Freezing for tasks requiring maximal comprehension retention (e.g., reading comprehension).

• Use Post-hoc Weight Reversion for tasks demanding superior cross-lingual transfer or high fluency (e.g., translation).

• Use Layer-Range L2 Regularization for balanced scenarios prioritizing language acquisition while maintaining robustness.

[What the improved AI can do]: The system will dynamically adjust its internal knowledge preservation mechanism, ensuring that a model fine-tuned for medical summarization prioritizes comprehension retention (via Freezing), while one fine-tuned for multilingual news translation prioritizes fluency (via Reversion).

)2. Develop Middle-Layer Drift Monitoring and Correction (The How):

Since the research identifies middle-layer drift as the primary driver of comprehension degradation, build a monitoring pipeline to detect this drift in real-time during continual pretraining.

[Improvement]: Integrate a layer-specific divergence metric that specifically tracks parameter movement in the middle layers (layers 9–27) relative to the base model initialization. When this drift exceeds a learned threshold, trigger an automatic soft regularization update (Layer-Range L2-SP) or a targeted post-hoc weight reversion on those specific layers.

[What the improved AI can do]: The system will proactively prevent catastrophic forgetting of general reasoning capabilities during language acquisition phases, ensuring that models remain robust across diverse languages even when learning new ones.

)3. Create Language Family Expert Architectures (The Structure):

Adopt the family-expert CPT paradigm as a core architectural template for multilingual LLMs, moving away from dense, monolithic training.

[Improvement]: Restructure the pretraining pipeline to train independent experts per language family (Slavic, Romance, Indic, etc.) rather than one giant model. Implement an efficient Model Soup mechanism that allows for on-demand merging of these experts when a new language is introduced or a specialized task requires cross-family knowledge integration.

[What the improved AI can do]: This will allow for massively scalable multilingual deployment. New languages can be added by training a dedicated expert, and the system can instantly leverage the combined knowledge of all existing family experts to handle complex, cross-lingual queries that require synthesized expertise (e.g., Translate this legal text from French into Marathi).

)4. Optimize Layer Sensitivity Based on Task Profile (The Refinement):

Leverage the causal analysis showing a mismatch between comprehension and translation layer sensitivities to fine-tune the model's output behavior for specific applications.

[Improvement]: For translation-heavy tasks, bias the post-training alignment towards restoring final-layer weights (as suggested by FLORES results). For reasoning/comprehension tasks, prioritize middle-layer freezing.

[What the improved AI can do]: The system will optimize its post-training phase not just for a single metric (like overall perplexity), but for the specific output behavior required—making it highly accurate at generating fluent, contextually appropriate translations while ensuring the underlying reasoning logic remains intact.

)5. Establish Cross-Lingual Evaluation Metrics (The Measurement):

Recognizing that perplexity is an incomplete proxy, mandate a suite of metrics that evaluate both acquisition and knowledge retention simultaneously.

[Improvement]: Replace reliance on single benchmarks with a multi-axis evaluation protocol: measuring perplexity on held-out languages (acquisition), Belebele/Global-PIQA accuracy (comprehension), and FLORES ChrF (translation quality) across all held-out relatives.

[What the improved AI can do]: The system will be rigorously benchmarked against these combined metrics, ensuring that improvements in one area (e.g., lower perplexity) are not masking degradation in another critical area (e.g., translation fluency).