Hybrid-Policy Self-Editing for Composable Unstructured Knowledge Editing

The paper introduces HPSE (Hybrid-Policy Self-Editing), a method designed to improve composability in Unstructured Knowledge Editing (UKE) for large language models (LLMs).

Problem: Composability Challenge in UKE

The paper identifies that existing UKE methods, which inject free-form passages containing multiple facts, fail to make the edited knowledge usable. The authors define composability as having two aspects:

• Decomposition: The ability to answer atomic questions about individual facts in the passage, rather than just recalling the whole passage.

• Composition: The ability to combine individual facts for multi-hop reasoning.

They introduce an untargeted regime where a generic editing prompt (e.g., Introduce X) is used, without specifying which facts are being updated. Benchmarking seven existing KE methods on two probes (UnKEBench for decomposition, MQuAKE-uns for composition) reveals that they consistently fall short of composability. The authors attribute this to editors' passive reliance on the provided editing context as the sole learning source, which induces severe memorization and poor generalization.

Proposed Method: HPSE

HPSE is a data-centric, plug-and-play approach that improves the training signal for gradient-based KE editors. It is built on on-policy self-distillation (OPSD), where the edited model (student) generates its own rollouts and is supervised by a privileged state of the same model (teacher) that reads the new passage in context. However, OPSD suffers from a coverage failure because the pre-edited model's rollouts rarely cover the novel injected knowledge.

To fix this, HPSE constructs hybrid rollouts by mixing token-level outputs from the student and the privileged teacher. The teacher steps in to place missing facts onto the student's trajectory when two conditions are met:

1. The privileged–student gap is large: log π⋆(y⋆t x, y τ

2. The privileged model is confident: π⋆(y⋆t x, y κ

The training objective is:

JHPSE(θ) = Jhybrid(θ) + λ JNLL(θ)

where Jhybrid is the KL-divergence distillation loss on the hybrid rollout, and JNLL is a standard negative log-likelihood anchor on the passage.

Theoretical Analysis

The paper provides a theoretical analysis showing that HPSE has a coverage and supervision advantage over pure OPSD. The informal theorem states: For an edit with a fact span of l novel tokens, the hybrid rollout visits every fact prefix... the hybrid signal is omega(l) while the OPSD signal is O(1), meaning that their ratio grows at least linearly with l. This advantage becomes more pronounced with longer fact spans.

Experimental Results

HPSE was evaluated on four LLMs (Qwen2.5, Qwen3, Llama3.1, Gemma2) and two gradient-based KE editors (FT-M and LoRA), and compared against five baselines (MEMIT, AlphaEdit, AnyEdit, UnKE, COIN⋆).

• Single Editing: HPSE provided consistent benchmark-average gains across all 16 editor–LLM–benchmark combinations. For example, FT-M gained +6.8 and +5.0 points on MQuAKE-uns and UnKEBench, respectively. HPSE improved both decomposition (Dmp.) and composition (Cmp.) scores while preserving locality (MMLU).

• Continual Editing: HPSE also improved performance when multiple edits were applied sequentially (T = 10, 20, 50, 100), with gains in 29 out of 32 settings.

• Ablations: Removing the hybrid-policy rollout (reducing to OPSD) degraded performance, confirming the coverage failure. Adding either the hybrid rollout or the NLL anchor improved OPSD, with the hybrid rollout being more impactful.

• Case Studies: HPSE successfully answered atomic questions directly (decomposition) and chained multiple edits for multi-hop reasoning (composition), while baselines either repeated the passage verbatim, hallucinated, or reverted to prior knowledge.

Conclusion

The paper concludes that HPSE provides a plug-and-play improvement for existing gradient-based KE editors, equipping them to inject knowledge that is both decomposable and composable. The authors plan to generalize HPSE toward lifelong and multimodal editing in the future.

Improvements to AI Systems:

1. Add a hybrid-policy self-editing module to gradient-based knowledge editors.

• The AI system can now inject new factual passages into LLMs without degrading the model’s ability to answer atomic questions about each fact (decomposition) or to chain facts across edits (composition).

• It will automatically switch between the student’s own token predictions and a privileged teacher’s predictions during training, only when the teacher is confident and the student is far off—preventing memorization of the passage and promoting generalization.

2. Implement on-policy self-distillation with coverage-aware rollout generation.

• The improved system generates training rollouts that explicitly include novel fact tokens by mixing teacher-forced tokens into the student’s trajectory.

• This ensures that even rare or unseen facts in the editing passage are visited during training, so the model learns to produce them in new contexts rather than ignoring them or repeating the passage verbatim.

3. Add a dual-objective training signal (hybrid distillation + NLL anchor).

• The system balances two losses: a KL-divergence distillation loss on the hybrid rollouts (for compositional generalization) and a standard negative log-likelihood loss on the original passage (for factual retention).

• This prevents catastrophic forgetting of the passage while actively teaching the model to recombine facts for multi-hop reasoning.

4. Enable plug-and-play integration with existing editors (e.g., FT-M, LoRA) without architectural changes.

• The improved system can be applied on top of any gradient-based knowledge editor, requiring only access to the editor’s training loop and the model’s token probabilities.

• It works across different LLM families (Qwen, Llama, Gemma) and scales to continual editing (up to 100 sequential edits) with consistent gains.

5. Add a confidence-gated teacher intervention mechanism.

• The system dynamically decides when to inject teacher tokens based on two thresholds: the privileged–student log-probability gap and the teacher’s absolute confidence.

• This reduces noise from low-confidence teacher predictions and prevents over-reliance on the teacher, preserving the student’s own learning signal.

What the Improved AI System Can Do:

• Answer atomic questions about facts from a newly injected passage (e.g., after reading “Einstein was born in Ulm and won the Nobel Prize in 1921,” it can answer “Where was Einstein born?” and “When did Einstein win the Nobel Prize?”) without needing the full passage as context.

• Perform multi-hop reasoning across multiple separate edits (e.g., after editing “Ulm is in Germany” and “Einstein was born in Ulm,” it can answer “In which country was Einstein born?”).

• Maintain existing knowledge (locality) while updating new facts, avoiding degradation on unrelated tasks like MMLU.

• Handle sequential edits without forgetting earlier edits, making it suitable for lifelong learning scenarios.

• Work with minimal hyperparameter tuning—the method is data-centric and does not require changing the model architecture or the editor’s core algorithm.

Large language models (LLMs) achieve remarkable performance across natural language tasks, yet they are trained on static corpora and their knowledge quickly becomes outdated in a fast-changing world. This motivates knowledge editing (KE), which updates specific knowledge in an LLM without changing unrelated others. Recent works move from structured knowledge triples toward unstructured KE (UKE), where the edit is a free-form passage that may state multiple facts at once. Nonetheless, existing editors inject such a passage yet fail to use it: the edited model can recall the passage, but can neither answer atomic questions about its facts nor compose them into multi-hop reasoning. We attribute this missing property, which we term composability, to editors' passive reliance on the fixed passage as the sole learning source. In response, we cast editing as a proactive self-distillation from a privileged in-context state of the same model, which requires no external supervision. We further reveal that due to the novelty of the injected knowledge, the pre-edited model's own rollouts rarely cover it, which limits the effectiveness of pure on-policy distillation. To close this gap, we propose HPSE, which builds a hybrid rollout that steps in to place missing facts onto the student's own trajectory precisely where its coverage fails, while staying on-policy elsewhere. We theoretically analyze HPSE's advantage over pure on-policy distillation, and empirically establish its plug-and-play improvements across four LLM backbones and two KE editors under various scenarios.