GROM: Gradient-Free Rapid One-Shot Machine Unlearning

The paper introduces GROM (Gradient-free Rapid One-shot Machine-unlearning), a fundamentally new approach to machine unlearning for large language models (LLMs) that replaces iterative, gradient-based fine-tuning with a direct, exact analytical solution. As the paper states: Instead of training additional parameters over multiple epochs, GROM computes a precise, additive update to targeted weight matrices in a single step. The core motivation is that current state-of-the-art approaches primarily rely on iterative, training-time unlearning via fine-tuning, which is computationally expensive and lacks explicit analytical formulations and can leave the targeted knowledge merely hidden rather than removed, to the point that simply quantizing the unlearned model restores much of what it was supposed to have erased.

GROM frames unlearning as a ridge-regularized least-squares optimization problem that forces a selected layer to suppress unwanted content while strictly maintaining its output on retained data. For a weight matrix W, the method computes an additive update P such that the new weights are W + P. The objective is:

min P∈R m×n (wr/r)‖P Xr‖2 F + (wf/s)‖P Xf − D‖2 F + µ‖P‖2 F

where Xf and Xr are per-token 'keys' collected from one gradient-free forward pass over the forget data and one over the retain data, and D is a prescribed unlearning target specifying the desired output-space displacement. The paper proves (Theorem 1) that this objective has a unique closed-form solution:

P⋆ = (wf/s) D XfT A−1, where A = (wr/r)XrXrT + (wf/s)XfXfT + µIn

This avoids iterative approximation entirely and requires no backpropagation and no iteration to convergence.

The paper specifies two target types:

1. Token suppression (for TOFU, MUSE): we steer the output away from the gold next token, using either one-hot vectors (for the LM head) or LM-head rows normalized (for hidden layers), with a specificity weight αj ∈ [0,1] that confines suppression to tokens distinctive of the forget set.

2. Representation corruption (for WMDP): with a single fixed random unit vector u ∈ Rm, we set dj = c u for all j, driving the layer's output on forget-like inputs toward a fixed, meaningless direction while leaving retain inputs intact.

GROM uses a logit-lens attribution score to identify layers that directly write the memorized forget tokens, computing Fl − Rl for each MLP layer l, where Fl measures the direct contribution of layer l to the logit of token gj on forget data and Rl is the analogous retain effect. A large positive Sl indicates that layer l contributes more to forget-specific gold-token logits than to retain-token logits. GROM then selects the contiguous window with the largest average score for a fixed edit width k.

A notable theoretical contribution is Theorem 2, which provides the deletion influence of example i in closed form using the Sherman–Morrison–Woodbury identity. Deleting a forget example perturbs the Gram matrix by a symmetric rank-ci downdate, enabling exact computation of each example's contribution to the edit with no n × n reinversion and no retraining. The paper notes: Gradient-based unlearning admits no analogue and must approximate this with influence functions or completely retrain the model per deleted example.

GROM is compared against gradient-based, representation-based, preference-based, and distillation-based approaches including GA, GradDiff, TaskVector, RMU, IDKDPO, IDKNLL, RKLD, UNDIAL, NPO, AltPO, and SimNPO, plus locate-then-edit knowledge editors (ROME, MEMIT, AlphaEdit, ZeroUnlearn). Evaluations use the open-unlearning protocol on TOFU-5%, TOFU-10%, MUSE-Books, MUSE-News, WMDP, and ZsRE, with wall-clock update times... on a single NVIDIA H100.

On TOFU-5% (LLaMA2-7B-Chat), GROM attains near-saturated forgetting while keeping utility close to the original model, giving the best Final Score and the highest MU (Final Score 0.79, MU 0.62, 1.8 minutes). On the harder TOFU-10% (LLaMA3.2-1B-Instruct), GROM avoids both failure modes—over-forgetting at the cost of utility (IDKDPO, UNDIAL) and preserving utility while under-forgetting (RMU)—with a single closed-form update, achieving Final Score 0.75 and MU 0.60 in 0.5 minutes. Notably, GROM achieves 1-Prob. of 1.00 and 1-Extraction Strength of 0.94 on the forget set.

On MUSE News (LLaMA2-7B), GROM attains the best Final Score (55.93 vs. 49.81 for SimNPO), driving privacy leakage close to zero (PrivLeak = −3.80). On MUSE Books (ICLM-7B), the harder entangled scenario, GROM achieves Final Score 76.20, preserv[ing] the highest retain utility (KnowMem Dr) while nearly eliminating verbatim memorization (VerbMem Df = 2.40) and driving privacy leakage to ≈ 0 (−0.22). Runtime is 0.3 minutes on both corpora, versus 18.0–62.5 minutes for gradient baselines.

On WMDP-Bio (Llama-3-8B-Instruct), GROM achieves by far the highest general-knowledge utility at competitive suppression, giving the best Final Score (0.63) with 1 − AccBio = 0.71 and MMLU = 0.55, with a single closed-form edit at one MLP layer in 0.2 minutes. The paper highlights that GradDiff... collapses MMLU to 0.26, while SimNPO reaches stronger suppression (0.75) but lower utility (0.44).

Tested inside the harness of (Lin et al. 2026) for fact unlearning, GROM outperforms these tuned editors: it removes targeted answers more effectively, generalizes better to paraphrases, and preserves locality at least as well, with head-only configuration.

A key result is that GROM is quantization-robust. Reproducing the finding that unlearning can be undone simply by quantizing the model, the paper shows that at 4-bit precision, SimNPO fails sharply, its verbatim memorization climbing from near zero back toward the original model on both corpora, whereas GROM does not recover, matching the gold Retrain model, which indicates that the closed-form edit removes the targeted content rather than hiding it in low-magnitude weights that rounding undoes.

• Specificity weighting: Disabling α lowers every score, confirming the specificity weight, and not suppression alone, is the component that preserves utility.

• PCA analysis: The closed-form patch moves forget logits substantially... while the retain logits remain nearly fixed.

• Edited matrix selection: Only the linear residual writers realize the edit, and among them GROM keeps the MLP writer, which is better or tied throughout. An ablation (Table 7) shows that disabling the specificity weight lowers performance on every benchmark.

• Layer selection: The selected band is therefore the most reliable choice, and it is the best variant on every benchmark, including against the wider edit that spends more layers to get there.

• Hyperparameter sensitivity: Only three hyperparameters are free (edit strength β, retain weight wr, ridge scale ρ); the results show β buys forgetting almost for free up to the tuned value, wr is the only axis whose two ends both fail outright, and ρ is a numerical safeguard rather than a tuning knob.

The paper's three main contributions are: (1) a Closed-Form Unlearning Formulation that completely bypass[es] the computational overhead and optimization instability of iterative fine-tuning; (2) Gradient-Free Efficiency making GROM up to two orders of magnitude faster than gradient-based approaches; and (3) State-of-the-Art Trade-offs achieving Pareto-best forgetting-utility trade-offs across five benchmarks while withstand[ing] the low-bit quantization attack. The authors conclude that GROM is to our knowledge the first one-shot unlearning method that achieves and surpasses the unlearning performance of iterative, optimization based methods, potentially pav[ing] the way to efficiently unlearning even larger models, where training based approaches are infeasible on limited hardware, and suggest an interesting avenue for future research to extend our method to the knowledge editing task. Code is publicly available at https://github.com/Batorskq/GROM.

Improved AI system capabilities enabled by GROM:

1. One-shot, exact unlearning without backpropagation.

Instead of iterative gradient fine-tuning over many epochs, the improved system computes a single closed-form weight update from one forward pass over forget data and one over retain data. It can remove targeted knowledge from LLMs in seconds to a few minutes (e.g., 0.3–1.8 minutes vs. 18–62 minutes for gradient baselines) on a single GPU, making unlearning feasible for models too large to retrain or fine-tune iteratively.

2. Quantization-robust knowledge erasure.

The improved unlearning system remains effective even after the model is quantized to 4-bit precision. Unlike gradient-based unlearning, where verbatim memorization resurfaces after quantization, this method deletes the actual content in the weight space rather than hiding it in low-magnitude weights. This is critical for deploying unlearned models on edge devices and mobile hardware.

3. Controlled forgetting with exact utility preservation.

The system can suppress forget-set-specific tokens while explicitly constraining retain-set outputs via ridge-regularized least squares. It achieves Pareto-best forgetting–utility trade-offs across TOFU, MUSE, WMDP, and ZsRE, with near-perfect forget-set privacy (e.g., 1-Prob. = 1.00, 1-Extraction Strength = 0.94) while keeping general knowledge (MMLU) and retain-set accuracy high.

4. Automatic layer selection via logit-lens attribution.

The system identifies which MLP layers directly write memorized tokens and edits only the minimal contiguous band of layers needed. This reduces collateral damage, preserves retain knowledge, and speeds up the update. It can decide exactly where to intervene, rather than editing all layers or relying on heuristic layer choices.

5. Exact per-example deletion auditing.

Because the update is a closed-form linear-algebraic operation, the system can compute the exact influence of deleting any single training example using Sherman–Morrison–Woodbury rank-1 downdates — no retraining, no influence-function approximation, no full Gram-matrix reinversion. This enables precise data-rights compliance auditing, such as how much did this specific user's data contribute to the unlearned content?

6. Head-only and layer-specific fact unlearning.

The improved system can operate in a head-only configuration for fact unlearning (ZsRE-style editing), removing targeted answers more effectively than locate-then-edit editors like ROME/MEMIT/AlphaEdit while generalizing better to paraphrases and preserving locality. It can also use a single fixed random direction for representation corruption (WMDP-style) to remove broad hazardous knowledge without collapsing general utility.

7. Faster iterative knowledge-editing workflows.

The same closed-form update can be extended to knowledge editing (as the authors suggest), allowing an AI system to insert or modify facts in one shot without gradient training. This enables real-time updating of deployed LLMs with new or corrected information, while keeping unrelated knowledge intact.

8. Hundreds of baselines beaten with minimal hyperparameter tuning.

The improved system has only three free hyperparameters (edit strength, retain weight, ridge scale), and two of them are robust — making it easy to deploy in practice. It outperforms GA, GradDiff, TaskVector, RMU, IDKDPO, IDKNLL, RKLD, UNDIAL, NPO, AltPO, and SimNPO on final scores and runtimes across multiple benchmarks.

Machine unlearning has become a critical capability for safely removing specific, sensitive knowledge from large language models (LLMs). Current state-of-the-art approaches primarily rely on iterative, training-time unlearning via fine-tuning. However, even when utilizing parameter-efficient dimensionality reduction techniques like LoRA, gradient-based optimization remains computationally expensive and lacks explicit analytical formulations. It can also leave the targeted knowledge merely hidden rather than removed, to the point that simply quantizing the unlearned model restores much of what it was supposed to have erased. To resolve this, we propose a novel one-shot unlearning approach, abandoning iterative optimization in favor of a direct, exact analytical solution. We frame the unlearning process as a ridge-regularized least-squares optimization problem, deriving a closed-form additive update for targeted weight matrices. This update forces the selected layer to suppress unwanted content while strictly preserving its behavior on retained data. Computed from gradient-free forward passes alone, with no backpropagation and no iteration to convergence, GROM applies the weight edit in mere seconds, which makes it orders of magnitude faster than traditional fine-tuning. Extensive evaluations demonstrate that GROM achieves state-of-the-art forgetting-utility trade-offs on TOFU-5%, TOFU-10%, MUSE-Books, MUSE-News and WMDP, significantly reducing computational overhead without sacrificing overall model performance. Because the update removes the targeted content from the weights instead of masking it, GROM also withstands the low-bit quantization attack that recovers much of the content a gradient-based baseline had appeared to forget. Our code is publicly available at https://github.com/Batorskq/GROM.