Exact Record Omission in Delta Attention: A Transport Criterion, Its Cost, and a Replay Certificate
cs.LG, cs.CL, cs.CR
Submitted: 2026-09-06
Updated: 2026-09-25
Comments: 22 pages, 8 figures, 9 tables. Source-free evidence for every quantitative claim is packaged with the source
License: http://creativecommons.org/licenses/by/4.0/
The gist: When a user asks an assistant to forget a record, the test is whether the memory now matches the state it would hold if the record had never been stored.
Terminology
Abstract
When a user asks an assistant to forget a record, the test is whether the memory now matches the state it would hold if the record had never been stored. Independently encoded rows can be removed directly; a recurrent memory folds records into an evolving state. One hope is a receipt: save the difference the record made when it arrived, carry it forward through later updates, and subtract it, so that deletion costs one fixed-size edit no matter how long the conversation runs. We show that a transported receipt reaches exact omission if and only if the changes the record induces in later updates cancel out on net, and we measure whether they do on the released 48B Kimi Linear hybrid. They do not: after 4,096 further tokens the record still leaves an imprint of about 4.5% of the state norm that none of the tested receipt classes removes, recomputing half the suffix closes less than half the gap, and the per-token log a receipt needs costs more than a full checkpoint after 88 tokens. The same write-rule classification held on Mamba-2, Falcon-H1, and RWKV-7 with predictions recorded before the runs. Restoring a checkpoint from before the record and replaying the surviving suffix matches the never-stored state exactly on every array we check. In the hybrid suffix sweep, masking the record's attention rows brings sampled recovery close to the never-stored floor even though the recurrent imprint remains, and an auditor who rebuilds the reference can still detect it. Among the evaluated methods, checkpoint replay achieves exact omission, with work proportional to the replayed suffix.
Sources
- Unlearning at Scale: State-Exact Trace-Preserving Deletion in Billion-Parameter Language Models
- Machine Unlearning
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- KVEraser: Learning to Steer KV Cache for Efficient Localized Context Erasing
- The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
- SnapKV: LLM Knows What You are Looking for Before Generation
- Eight Methods to Evaluate Robust Unlearning in LLMs
- TOFU: A Task of Fictitious Unlearning for LLMs
- Can an AI Assistant Really Forget? Auditable Deletion from Addressable Memory
- What a Deletion Certificate Covers, and Where It Expires: Auditable Removal from a Support-Vector Memory
- MUSE: Machine Unlearning Six-Way Evaluation for Language Models
- Position: LLM Unlearning Benchmarks are Weak Measures of Progress
- Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
- Parallelizing Linear Transformers with the Delta Rule over Sequence Length
- Gated Delta Networks: Improving Mamba2 with Delta Rule
- VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
- Kimi Linear: An Expressive, Efficient Attention Architecture
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks