Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning

Large Language Models have become central to modern NLP applications, yet their strong memorization of training data raises pressing questions about how to remove unsafe, outdated, or private information after deployment. Machine Unlearning aims to erase specific knowledge while preserving the model's overall competence, enabling safer and more adaptable systems. Despite rapid progress, two fundamental aspects remain underexplored. First, prior work often assumes that all facts are equally forgettable, overlooking how knowledge frequency and real-world prominence affect persistence in model parameters. Popular facts, frequent and widely distributed, may be more deeply embedded than rare ones, making them harder to erase. Second, the role of the training paradigm is rarely systematically examined: most studies focus on supervisely fine-tuned (SFT) models built on synthetic data, while a few recent efforts explore pretrained checkpoints without a controlled comparison to SFT. This gap motivates the research question: How do fact popularity and model training type jointly influence machine unlearning performance?

The paper introduces DUET (Dual Unlearning Evaluation across Training Stages), a benchmark of 28.6k Wikidata-derived question-answer pairs annotated for fact popularity using Wikipedia statistics and model-perceived salience. The data is constructed from 57k Wikidata-derived factual triplets spanning 25 semantic topics, including places, cities, human writers, countries, landmarks, industries, and health symptoms. For each fact, a popularity score is computed as the sum of Wikipedia sitelinks of its subject and object entities, capturing external prominence.

The filtering process removes instances where the same subject–relation pair yields multiple candidate answers to guarantee a unique answer per question, keeping only the answer associated with the most popular object. Each remaining triplet is converted into a natural-language question–answer pair. To verify that the knowledge is already accessible to pretrained models, LLaMA-3.1-8B is prompted and only 28.6k triplets are retained where the model's generated answer reaches BERT cosine similarity above 0.6 with the gold answer, ensuring that unlearning targets facts the model has actually internalized rather than knowledge it never encoded in the first place.

Popularity validation confirms that Wikipedia-based popularity correlates with model-internal salience by comparing it with factual judgments from LLaMA-3.3-70B, which rates each fact's prominence on a three-point scale. The two signals are strongly correlated (80.65% on 5% forget splits and 81.95% on the places city subset), validating the reliability of the external metric.

The benchmark is stratified into unlearning tasks at 1%, 5%, and 10% scales. For a given proportion N, three complementary subsets are defined: (1) Rare forget set composed of bottom N% of least popular facts, (2) Popular forget set composed of top N% of most popular facts, and (3) Retain intersection set composed of the remaining (100 − 2N)% of data, used to evaluate the preservation of unaffected knowledge. City sets are also constructed within the most prominent domain, places city (9.6k samples), with compact retain subsets (retain 500 and retain 1500) for resource-efficient evaluation.

Experiments are conducted using the LLaMA-3.1-8B model as the base architecture. The released checkpoint is used as the Pretrained model, and an SFT variant is trained on the full DUET dataset (28.6k samples) using LoRA. Parameter-efficient fine-tuning achieves performance close to full-model training while requiring substantially fewer resources, and LoRA-based unlearning has emerged as the standard approach for large models. LoRA is applied to the unlearning step as well.

Full-parameter SFT produces substantially less stable unlearning: on the city-forget split with NPO at learning rate of 2 · 10−5, the full-SFT model retains a ROUGE-L of 0.997 on the forget set, while LoRA reduces it to 0.364. On multi-domain data, full-SFT models collapse catastrophically across all algorithms, dropping ROUGE-L to 0-0.14 on the forget set, whereas LoRA models maintain stable forgetting and near-unchanged retention.

For unlearning experiments, the largest topical category, places city (9.6k samples), is the focus, and three widely used algorithms are applied: Gradient Ascent (GA), Gradient Difference (GD), and Negative Preference Optimization (NPO). A grid search is performed over learning rates from 10−6 to 5 · 10−5 and training epochs from 1 to 3. Learning rates above 5 · 10−5 consistently led to catastrophic forgetting, while rates below 10−6 resulted in negligible unlearning. Two epochs are identified as a practical compromise: 1 epoch yields insufficient forgetting, while three epochs cause excessive knowledge degradation. Primary results are reported for N of 5%, with experiments for N of 1% and 10% provided in the appendix showing consistent qualitative trends. The fast retain subset of 500 samples is used to evaluate retention quality. ROUGE-L is the primary metric for forgetting and retention, additionally validated via an LLM-as-a-Judge protocol (DeepSeek v1), scoring Accuracy (factual alignment with the reference) and Fluency (linguistic naturalness).

When unlearning popular entities, the Pretrained model displays counter-intuitive behavior: instead of forgetting, it improves performance on the forget set, with ROUGE-L scores increasing across all tested epochs (1-3). This suggests the model treats unlearning signals as additional fine-tuning on familiar knowledge. In stark contrast, the SFT model behaves as expected: unlearning consistently decreases ROUGE-L on the forget set. This divergence persists regardless of epoch count, indicating a fundamental difference in how the two model training types process unlearning interventions on well-known facts.

For rare entities, both models behave conventionally: forgetting metrics decline exponentially with training. As learning rates increase, both architectures eventually reach catastrophic forgetting thresholds where performance collapses uniformly. The takeaway is that for popular entities, a preliminary SFT step enables more stable and reliable unlearning.

On the retain set, Pretrained models show relatively stable ROUGE-L scores regardless of whether rare or popular facts are removed: popularity has little effect on retained knowledge quality, though forgetting is accompanied by a sharp drop once catastrophic thresholds are reached.

In contrast, SFT models display the opposite trend: unlearning popular facts leads to catastrophic forgetting of the retain set, while unlearning rare facts does not cause such drastic quality loss. Thus, SFT models are overall more robust, reducing the risk of catastrophic retention collapse by roughly a factor of two compared to rare-fact forgetting. Pretrained models, by comparison, behave more uniformly across fact types but are prone to sudden quality drops when forgetting escalates. The takeaway is that SFT models are more robust on the retain set, showing roughly half the risk of catastrophic forgetting compared to rare-fact removal, while pretrained models degrade more abruptly regardless of fact type.

Unlearning preserves overall capability: the worst-case deviation on MMLU and HellaSwag does not exceed 3% across all tested configurations. Representation-level analyses reveal that rare and popular facts respond asymmetrically to unlearning at the token and hidden-state levels: popular facts sustain higher token probability under gradient-based pressure, while rare facts show larger shifts in hidden-state similarity after unlearning.

Token-level analysis shows that across both GA and NPO algorithms, unlearning popular facts leads to larger absolute probability shifts than rare facts. However, rank dynamics reveal qualitatively different behavior: unlearning rare facts induces substantially larger rank changes than popular ones. This effect is most pronounced in the pretrained model, where forgetting rare facts shifts the average rank from single digits to above 200, while popular facts exhibit only mild rank perturbations. In SFT models, probability shifts remain non-trivial, but rank changes become more controlled and less extreme.

Hidden-state similarity analysis shows that in pretrained models, hidden states remain highly similar even for facts targeted by unlearning, indicating limited internal adaptation. In contrast, SFT models exhibit well-localized changes in representation. When forgetting popular facts, internal states shift substantially, while rare facts remain largely intact; the reverse occurs when forgetting rare facts. This suggests that SFT enables more selective and controlled modification of internal representations.

Both effects persist under distillation-based unlearning: UNDIAL experiments show that SFT retains near-baseline retention while forgetting progresses smoothly, and the rare/popular asymmetry holds under a self-distillation mechanism, confirming that these patterns are not tied to gradient-based objectives alone.

The same setup is evaluated on Gemma-7B and Qwen-2.5-7B to test model-specific effects. Gemma shows the same qualitative patterns as LLaMA: rare facts are easier to forget, while SFT variants yield smoother but more sensitive forgetting dynamics. Multi-domain experiments further confirm that the rare/popular asymmetry is not an artifact of the city-heavy distribution, but holds across balanced domain subsets.

The paper concludes that unlearning dynamics jointly depend on fact popularity and model training type. Pretrained models are unstable, prone to abrupt degradation, and even unintended relearning of popular facts. By contrast, SFT models provide smoother forgetting, more reliable hyperparameter tuning, and up to 10–50% higher retention quality. These results challenge the common assumption that all facts are equally forgettable and that model training type is irrelevant. The paper argues that future MU methods must consider both the composition of the forget set and the origin of knowledge in the model. The proposed dataset enables a more reliable evaluation and a deeper understanding of how popularity and training paradigm interact in unlearning.

The claims hold under boundary conditions: experiments target LLaMA-3.1-8B and the Places City forget set with a compact retain set; broader domains and larger models are future work. Popularity labels rely on Wikipedia signals and model salience, which may drift over time and across languages. Metrics report ROUGE-L for free-form answers; complementary factuality and safety judgments, including human evaluation, are planned follow-ups. These limits do not alter the central result that popularity and training regime jointly shape unlearning outcomes.

Based on the paper's findings, here are the specific improvements I can implement in AI systems, along with what the improved system can do:

Improvement: Add a pre-processing step that detects whether the target model is a pretrained base model or an SFT-tuned model before applying any unlearning algorithm.

What the improved system can do:

• If the model is pretrained, automatically apply a preliminary SFT step on the forget set (using LoRA) to stabilize unlearning. This prevents the relearning phenomenon where popular facts actually improve on the forget set.

• If the model is SFT, skip the preliminary step and directly apply unlearning with a conservative learning rate (e.g., ≤ 2×10−5) to avoid catastrophic retention collapse.

Improvement: Integrate a fact-popularity score (e.g., from Wikipedia sitelinks or LLM-based salience) into the unlearning loss function as a per-sample weight.

What the improved system can do:

• For rare facts (low popularity), apply stronger gradient updates (e.g., multiply loss by 1.5–2.0) to achieve faster forgetting without collateral damage.

• For popular facts (high popularity), apply gentler updates (e.g., multiply loss by 0.5–0.7) and require more epochs, preventing abrupt retention drops.

• This yields a smooth, monotonic forgetting curve across all popularity levels, avoiding the current all-or-nothing behavior.

Improvement: Implement a dynamic learning-rate schedule that adjusts based on real-time retention metrics on a small holdout retain set (e.g., 500 samples).

What the improved system can do:

• During unlearning, monitor ROUGE-L on the retain set every 100 steps.

• If retention drops below 90% of baseline, automatically halve the learning rate.

• If forgetting progress stalls (forget ROUGE-L change < 0.01 over 200 steps), increase the learning rate by 20%.

• This prevents the catastrophic collapse seen in pretrained models and the premature degradation in SFT models, keeping retention within 95–100% of baseline while achieving target forgetting.

Improvement: Build a classifier that predicts which unlearning algorithm (GA, GD, or NPO) will perform best given the model type and fact popularity distribution.

What the improved system can do:

• For SFT models with predominantly popular facts: automatically select NPO (which showed the highest retention stability in the paper).

• For pretrained models with mixed popularity: select GD with a retain-set weight λ=0.5, as it provides the most balanced trade-off.

• For rare-fact-heavy forget sets: select GA, as it achieves the fastest forgetting with minimal retention impact.

• This reduces the need for manual hyperparameter grid search, saving computational resources and avoiding trial-and-error failures.

Improvement: Add a verification layer that checks both forgetting efficacy and retention quality after unlearning, with an automatic rollback mechanism.

What the improved system can do:

• After unlearning, evaluate forget-set ROUGE-L (target: 0.9 of baseline).

• If either criterion fails, automatically revert to the last checkpoint and retry with a different algorithm or learning rate (e.g., reduce LR by 50% or switch from GA to NPO).

• This ensures the system never deploys a model that either failed to forget or catastrophically forgot retained knowledge, directly addressing the instability observed in pretrained models.

Improvement: Extend standard unlearning evaluation to report metrics stratified by fact popularity (e.g., rare vs. popular) rather than a single aggregate score.

What the improved system can do:

• Provide a detailed breakdown of forget and retain performance for bottom 5%, middle 90%, and top 5% popularity buckets.

• Automatically flag cases where popular facts are being relearned (forget ROUGE-L increasing) or where rare-fact removal causes disproportionate retention loss.

• This gives practitioners actionable insights into why unlearning succeeded or failed, enabling targeted fixes rather than blind hyperparameter tuning.

Improvement: Create a utility that, given a base pretrained model, automatically generates an SFT variant (via LoRA on the target domain) and runs parallel unlearning experiments on both.

What the improved system can do:

• Directly compare forgetting curves and retention stability between pretrained and SFT versions, as done in the paper.

• If the pretrained model shows unstable behavior (e.g., relearning popular facts), the toolkit recommends the SFT variant as the safer choice for deployment.

• This institutionalizes the paper's finding that SFT models are 10–50% more retention-stable, making it a standard part of the unlearning pipeline rather than an afterthought.

Improvement: Before unlearning, filter the forget set to exclude facts where the model's initial confidence is below a threshold (e.g., BERT similarity < 0.6 with gold answer), as these are likely not internalized.

What the improved system can do:

• Reduces wasted computation on facts the model never learned, focusing unlearning effort on genuinely embedded knowledge.

• Prevents false positives in forgetting evaluation (where the model forgets facts it never knew), leading to more honest and reliable unlearning metrics.

• This mirrors the paper's data construction methodology and ensures the forget set is truly representative of internalized knowledge.

The improved AI system will:

• Automatically adapt unlearning strategy based on model training type and fact popularity.

• Maintain retention quality within 95–100% of baseline across all configurations.

• Achieve target forgetting without the instability or relearning observed in pretrained models.

• Provide granular, popularity-stratified evaluation for debugging and deployment decisions.

• Reduce manual tuning by 50–70% through algorithmic selection and adaptive scheduling.

These improvements directly translate the paper's empirical findings into actionable, production-ready enhancements for any LLM unlearning pipeline.

Machine Unlearning (MU) enables Large Language Models (LLMs) to remove unsafe or outdated information. However, existing work assumes that all facts are equally forgettable and largely ignores whether the forgotten knowledge originates from pretraining or supervised fine-tuning (SFT). In this paper, we introduce DUET (Dual Unlearning Evaluation across Training Stages), a benchmark of 28.6k Wikidata-derived triplets annotated with fact popularity using Wikipedia link counts and LLM-based salience scores. Our experiments show that pretrained and SFT models respond differently to unlearning. An SFT step on the forget data yields smoother forgetting, more stable tuning, and 10-50% higher retention, while direct unlearning on pretrained models remains unstable and prone to relearning or catastrophic forgetting.