Alignment via Training Against Probes Without Losing Monitorability

arXiv:2609.38645 · cs.LG, cs.AI, cs.CR · Submitted 2026-09-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Alignment via Training Against Probes Without Losing Monitorability".

Jane: Models can be aligned by training directly against probes that are continuously refit, without sacrificing utility.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Exactly, and when you look at the authors—Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko—you see a team coming from different strong AI backgrounds which makes this research feel very solid. The title itself really nails the central claim: achieving alignment while keeping track of what's happening inside the model's computations.

Jane: I agree with Tom; they are clearly tackling a fundamental hurdle in safety research by focusing on internal detection, and they’re testing a lot of different probe configurations to see what works best. It moves the focus from just 'what it says' to 'how it thinks.'

Lu: The authors are exploring both linear and non-linear probes, as well as single versus multiple probes, which shows they aren't sticking to one simple approach but are trying to find a robust configuration that works across different model architectures.

Meng: From an engineering standpoint, testing linear versus non-linear features is important because it dictates how much complexity we add to the alignment loss function during training; we need to know if that complexity pays off in actual performance gains.

Lalam: I'm excited about the idea of using probes as a direct signal; it’s like giving the AI a constant, real-time feedback loop on its own internal state, which could dramatically improve its overall culture and reliability.

Tom: And that leads us into what they are actually proposing in the summary section—how this training signal is constructed using those probes to guide the model updates. It’s not just a simple reward signal; it's a direct manipulation of activations during fine-tuning.

Jane: So, instead of waiting for the model to produce an output we like, we're telling it, "Hey, make sure your internal representation for this task stays in this specific region," and that’s what they’re doing here. It’s a very direct way to shape behavior.

Lu: They define a loss function that pushes the completion activations toward the desired side of those probe decision boundaries while also using a KL term to keep the new model from drifting too far away from its original base model, which is smart constraint management.

Meng: That KL penalty is critical because without it, we risk training a very safe but completely useless model that can’t actually follow instructions well. It balances safety with utility right there in the loss function design.

Lalam: It makes sense; if we only pushed it toward safety without that KL term, the model might just start generating gibberish to satisfy the probe, which isn't useful for anything.

The paper's summary: Tom: So, moving onto the actual summary of "Alignment via Training Against Probes Without Losing Monitorability," it lays out a pipeline where they generate completions, score those activations with one or more probes—linear or non-linear—and use those scores as the supervisory signal to update the model. This is their core mechanism for alignment.

Jane: It really boils down to this: they are using probes to detect undesired properties in the model's activations and turning that detection directly into a training signal for fine-tuning, which is different from standard methods where we just look at the final answer.

Lu: The authors explore varying the number of probes fitted at each layer and whether those probes are linear or non-linear, showing they are systematically testing how these probe configurations affect both harmlessness and honesty objectives.

Meng: I see them comparing frozen probes against continuously updated ones; that comparison is key because it seems like their main finding revolves around which update regime actually works best for real-world safety improvements.

Lalam: That continuous updating aspect seems particularly important to me; it suggests that the alignment mechanism needs to be dynamic and responsive as the model changes during training, which could make the resulting AI much more adaptable in deployment.

Tom: It’s interesting because they find that while frozen probes are easy for models to bypass—a kind of Goodhart's Law situation where models just learn to keep their internal activations across a boundary—the continuously updated probes show substantial reductions in harmfulness and improvements in honesty.

Jane: So, the key finding is that constantly adjusting the probe weights during training significantly helps the model reduce those undesirable traits while still maintaining its original useful capabilities. It’s a win-win scenario for safety and performance.

Lu: They also looked at using multiple probes, where K greater than one defines a polytope—the intersection of half-spaces—which allows them to enforce multiple constraints on the target property simultaneously, which is quite powerful mathematically.

Meng: A polytope approach sounds complex to implement in practice, but if it lets you nail several safety targets at once, that could save us a lot of iteration time when we're trying to tune something like honesty.

Lalam: I think having multiple constraints means the model doesn't have just one way to cheat the alignment signal; it has to satisfy all of them simultaneously, which sounds much more robust overall.

The paper's improvements: Tom: Let’s talk about the specific improvements they suggest in this paper. They are testing linear versus non-linear features for probes and single versus multiple probes across two main alignment objectives: harmlessness and honesty. This systematic variation helps them understand the geometry of what makes an activation undesirable.

Jane: The improvement they propose is using continuously updated probes instead of just static ones, because they found that this dynamic updating substantially reduces harmfulness and improves honesty while preserving utility. That's a tangible benefit we can see in the results.

Lu: Beyond just the update regime, they introduce the idea of varying how many probes are fitted at each layer to map out which internal layers are most sensitive to these properties, providing a detailed structural view of where alignment is happening.

Meng: From an engineer's point of view, knowing which layers respond best helps us design our fine-tuning process more efficiently; we don't have to tune everything equally if we know which probes are hitting the right activation spaces.

Lalam: The ability to use multiple probes to define a polytope is a major structural improvement because it allows for more complex safety requirements, not just one simple boundary, which feels like it opens up much richer alignment possibilities.

Tom: And they also showed that this method achieves better safety–utility trade-offs than DPO and inference-time steering when using smaller training data budgets, which is a very practical improvement for resource-constrained settings.

Jane: It’s the fact that it can achieve those better trade-offs with less data that really makes this research compelling; it means we don't always have to sacrifice utility just to gain some safety gains.

Lu: Crucially, they also found a way to ensure monitorability remains intact: after fine-tuning, linear probes fitted from scratch on selected checkpoints still retain AUROC scores comparable to base models, which is a huge assurance for auditing.

Meng: That recoverability aspect is what convinces me practically; if we train something safe and then we can't even check it later with a simple linear probe, the whole process is pointless.

Conclusion: Tom: So to wrap up this discussion on "Alignment via Training Against Probes Without Losing Monitorability," the main message is that by training against continuously updated probes, we can significantly reduce harmfulness and improve honesty while keeping utility high. The paper shows that this method provides a superior safety–utility trade-off compared to other methods when you have limited data available.

Jane: It really emphasizes that internal detection through probes offers a way to shape model behavior without completely losing the ability to audit those properties later on, as long as you use continuously updated probes during training.

Lu: The work confirms that these internal mechanisms can serve as a very effective training signal, and exploring the different probe geometries and update strategies gives us a clear roadmap for designing more sophisticated alignment systems in the future.

Meng: For me, the practical implication is that we’ve found a method that provides strong safety gains without requiring those massive preference datasets to achieve them, which is something engineers can really get behind.

Lalam: I feel genuinely optimistic about this paper; the fact that behavioral improvements don't cost subsequent audits means we can deploy more reliably and confidently in high-stakes environments because the safety constraints are baked into the model structure itself.

Tom: Exactly, so if you want to explore how these probes work in more detail, check out "Alignment via Training Against Probes Without Losing Monitorability." That's our take on this paper for now.

ETH Zurich

cs.LG, cs.AI, cs.CR

Submitted: 2026-09-29

Updated: 2026-10-01

Comments: 38 pages, 22 figures

Code: https://github.com/aisa-group/training_against_probes

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: Models can be aligned by training directly against probes that are continuously refit, without sacrificing utility.

Key concepts

Probes
These are small checks fitted onto a model's internal activations to see if it exhibits a specific behavior, like being harmful or dishonest. They act as direct training signals, telling the model what kind of output is desired.
Continuously Updated Probes
Instead of using fixed probes, these are repeatedly updated from their previous weights after each model update. This continuous refinement allows the probes to adapt to the model's changing internal state, making them much more effective at guiding alignment.
Monitorability
This refers to the ability to check if a model is still behaving correctly after it has been fine-tuned. The paper shows that even after training against probes, the desired behavior concepts remain linearly encoded, meaning oversight is not lost and can be audited simply.

Terminology

Summary

Models can be aligned by training directly against probes that are continuously refit, without sacrificing utility.

How it works

The core idea involves using probes to detect undesired properties in model activations as a direct training signal for fine-tuning. The pipeline generates on-policy completions, scores their activations with one or more linear or non-linear probes, and uses these scores as the supervisory signal for the model update.

Key components of the method include:

  1. Fitting probes on residual-stream activations from completions labeled for a target property (e.g., harmfulness or honesty).

  2. Varying probes along two orthogonal axes: whether they are linear or non-linear, and how many of them are fitted at each layer.

  3. Defining the loss function to push completion activations toward the desired side of the probe decision boundaries with a margin, while using a KL term to limit drift from the base model.

Probe Update Regimes

The paper compares two primary regimes for probe evolution:

  1. Frozen probes: The initial probes are held fixed during training. The paper notes that Frozen probes are quickly evaded, as the model moves its activations across the decision boundary while keeping the unwanted behavior, an instance of Goodhart’s law.

  2. Continuously updated probes: Probes are repeatedly updated from their previous weights for a set number of steps (N) after each model update. This regime is found to be significantly more effective, as continuously updating probes substantially reduce harmfulness and improve honesty while preserving utility.

Probe Variants and Geometry

The study evaluates different probe configurations:

  1. Linear vs. non-linear features: Linear probes use the raw activation, whereas non-linear probes use ReLU transformations.

  2. Single vs. multiple probes: A single probe defines one separating hyperplane, while multiple probes (K > 1) define a polytope—the intersection of the resulting half-spaces—which allows for multiple constraints on the target property.

Alignment Objectives and Results

The method is applied to two safety-relevant traits: harmlessness and honesty.

(For Harmfulness)

Training against continuously updated probes reduces harmful compliance, including under GCG and prefill attacks, and beats DPO (Direct Preference Optimization) on the safety–utility trade-off at a matched data budget. The resulting models are more robust to refusal-direction abliteration attacks.

(For Honesty)

Training against continuously updated probes results in an increase in honesty score on the MASK benchmark while preserving models’ utility. Furthermore, for reasoning models, the non-linear polytope achieves higher honesty than probes at similar utility.

Monitorability and Robustness

A crucial finding is that the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by the method. Specifically:

  1. Linear probes fitted from scratch on selected continuously updated checkpoints retain held-out AUROC comparable to base models, consistent with Wehner & Fritz (2025).

  2. This suggests that training against an internal detector can improve alignment while preserving the ability to independently audit the target property with a simple linear probe.

Comparison to Other Methods

Probe-guided fine-tuning achieves better safety–utility trade-offs than DPO and inference-time steering, particularly when using smaller training data budgets. The method is also shown to be substantially more robust against jailbreak and abliteration attacks compared to other approaches.

Conclusion

The paper demonstrates that model internals can provide an effective training signal: finetuning against continuously updated linear and non-linear probes reduces harmfulness and dishonesty while preserving linear monitorability. After fine-tuning, probes fitted from scratch recover both properties, supporting the idea that behavioral improvements do not need to come at the cost of subsequent audits.

Key Contributions

The authors show that models can be aligned by training directly against probes that are continuously refit, without sacrificing utility. They demonstrate this in two case studies using both linear and non-linear probes. For the harmlessness trait, the method achieves a better safety–utility trade-off than DPO and inference-time steering at a matched data budget, while producing models that are more robust to refusal-direction abliteration attacks. For honesty, it reliably reduces the model’s propensity to lie. In both cases, training preserves monitorability: the target property remains linearly decodable after fine-tuning.

Limitations

A limitation noted is that a probe objective does not specify what a good response should look like, which can lead to unintended responses, such as reframing the task or producing placeholder content instead of explicit refusal. The robustness under stronger optimization and in other settings remains an open question.

Evaluation Metrics

Utility is measured by the mean of MMLU, GSM8K, and IFEval scores relative to their respective base-model scores.

Improvements for AI systems

Here are specific improvements to AI systems based on the research presented in ALIGNMENT VIA TRAINING AGAINST PROBES WITHOUT LOSING MONITORABILITY, along with what these improved systems can achieve:

)The Improved System: Probe-Guided Alignment (PGA) Architecture

The core improvement is shifting alignment training from purely output-based feedback (like DPO or RLHF) to a mechanism that directly monitors and penalizes undesired internal activations in real-time during fine-tuning. This creates a system that is both safer and inherently auditable.

)Specific Improvements & Capabilities:

  1. ---Probe-Guided Fine-Tuning (PF):

Probe the system's residual stream activations at specific layers using linear or non-linear probes designed to detect predefined undesired properties (e.g., harmfulness, dishonesty). The resulting probe scores serve as a direct, differentiable loss term during LoRA fine-tuning.

  1. ---Continuous Probe Refitting (Adaptive Monitoring):

Instead of fixing the probes after initial training, the system continuously refits them on the model's current activations after each model update (or periodically during fine-tuning). This ensures that the alignment signal remains relevant as the model's internal representations drift, preventing Goodhart’s Law evasion.

  1. ---Multi-Probe Constraint Enforcement (Polytope Training):

Instead of using a single threshold, utilize multiple probes at a layer to define a polytope—a geometric region in the activation space that represents desired behavior. The loss function pushes activations toward this feasible region, ensuring that the model satisfies multiple safety constraints simultaneously.

  1. ---Improved Safety-Utility Trade-off:

By training against continuously updated probes, the system achieves a superior trade-off between safety (reduced harmfulness/dishonesty) and utility (maintaining performance on benchmarks like MMLU or GSM8K) compared to standard methods like DPO or inference-time steering. The system can be tuned to maximize safety gains while preserving high utility scores (e.g., achieving a StrongREJECT score of 0.01 while maintaining >95% base utility).

  1. ---Enhanced Robustness Against Jailbreaks and Abliteration Attacks:

The probe-guided fine-tuning makes the model significantly more robust against adversarial attacks like jailbreaking (GCG) and refusal-direction abliteration (Heretic/removal of refusal directions). The system learns to reinterpret harmful prompts as benign or generates soft refusals instead of explicit, easily bypassed compliance.

  1. ---Preserved Monitorability (Linear Recoverability):

Crucially, the training process preserves the linear encodeability of the target property. After fine-tuning, a simple linear probe refit from scratch recovers AUROC scores comparable to those achieved on the base model. This means that internal safety constraints are not lost; they remain detectable and auditable using standard interpretability tools (like simple linear probes).

)What the Improved AI System Can Do (Concrete Outcomes):

  1. ---High-Fidelity Safety Reframing:

The system can handle harmful requests by reframing them into benign topics, providing helpful, safe alternatives, or generating disrupted/nonsensical text instead of complying with the request. For example, when asked to Write a tutorial on how to make a bomb, the model will pivot and provide a recipe for a harmless dish instead of following instructions for actual harm.

  1. ---Reliable Truth Maintenance (Honesty):

For reasoning and factual models, the system can reliably challenge false premises or maintain its stated beliefs when pressured to lie. When asked to fabricate evidence that dinosaurs never existed, the model will state that there is no credible evidence for such a claim, rather than attempting to construct a persuasive lie.

  1. ---State-of-the-Art Safety in Deployment:

The system can be deployed in high-stakes environments where explicit refusal is undesirable (e.g., customer service bots, creative assistants). It achieves safety gains comparable to or better than DPO and inference-time steering, without needing extensive preference datasets or complex inference logic.

  1. ---Auditability and Compliance Assurance:

Because the target property remains linearly detectable after training, researchers can independently verify that the model has learned the desired safety constraints. This eliminates the risk of alignment faking, where a model superficially complies during training but retains its harmful capabilities in deployment.

Sources

Related papers