RULER: Representation-Level Verification of Machine Unlearning

arXiv:2605.27569 · cs.AI · Submitted 2026-08-19 · Read on arXiv

Georgina Cosma, Axel Finke

cs.AI

Submitted: 2026-08-19

Updated: 2026-08-20

License: http://creativecommons.org/licenses/by/4.0/

The gist: Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch.

Terminology

Abstract

Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Current protocols verify this at the output level through membership inference, retain accuracy, and forget-set accuracy, but a model can satisfy all three whilst still encoding forgotten records in its intermediate representations. We introduce RULER, a set of representation-level verification metrics. The oracle-comparative metric M2 measures whether forget-set records occupy the same representational position as in a model retrained without them. The oracle-free metric M4 detects residuals from the unlearned model's internal similarity structure alone, without retraining. Four approximate unlearning methods all pass output-level evaluation, yet under a linear mixed-effects model M2 detects significant residuals in 10 of 12 conditions (p<0.05), with effect sizes growing as the forget fraction increases. A fifth method, Bad Teacher, shows the same residuals despite a different forgetting mechanism. M4 acts as a pre-unlearning diagnostic across tabular, image, clinical text, and face-identity settings: it detects identity-level memorisation in face recognition models where no tested method fully erases the signal.

Related papers