Steering the Language Axis: From Linear Decodability to Causal Control

arXiv:2608.12334 · cs.CL, cs.AI · Submitted 2026-06-02 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Steering the Language Axis: From Linear Decodability to Causal Control".

Jane: The paper was written by Arnav Srivastav from University of California, Santa Cruz.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the arXiv radio hour, everyone. I'm Tom, and joining me as always is the brilliant Jane. Today we're digging into a paper that's got a title that just grabs you: "Steering the Language Axis: From Linear Decodability to Causal Control."

Jane: Tom, that title is doing a lot of work. It's basically saying, "Hey, we know we can read the language a model is thinking in, but can we actually grab that dial and turn it?" And that's a huge jump.

Tom: Right, and for our listeners who might not be deep in the weeds, let's break that down. For years, researchers have shown you can look at the internal activations of a model and decode what language it's using. That's the "linear decodability" part. It's like reading the label on a box.

Jane: But reading the label doesn't mean you can change what's inside the box. That's the "causal control" part. This paper from UC Santa Cruz, by Arnav Srivastav, is asking whether that label is just a sticker or if it's actually connected to the machinery that decides what comes out.

Tom: And the answer they found is pretty wild. They didn't just find a dial; they found a very specific dial that works differently depending on where you touch the model. We're talking about models like Qwen three point five-2B and Llama-three point two-1B.

Jane: So they're not just looking at one model and hoping it generalizes. They're testing across architectures, which is a big deal for any kind of mechanistic claim. And they're testing across language pairs, like English to Chinese, which is a completely different script, and English to Spanish, which shares the same alphabet.

Tom: Exactly. And the implication here is massive. If you can causally control language, you're not just doing a parlor trick. You're potentially fixing a real problem where models just default to English or mix languages when you don't want them to.

Jane: It's like having a volume knob for language instead of having to rewire the entire stereo system every time you want to change the station. That's the promise, anyway. But as we'll see, it's not that simple.

Tom: Not even close. Because they found that this "language axis" isn't a single universal switch. It's more like a series of localized control panels scattered throughout the network, and they all behave differently.

Jane: And that's where the real science starts. We're going to get into the weeds of how they found this axis and why some layers of the model just refuse to cooperate. Stick around.

Summary: Tom: So, Jane, we've got the title unpacked. Now let's talk about what this paper actually did. The core idea in "Steering the Language Axis" is that they took these massive models and tried to find a single direction in the mathematical space of activations that corresponds to "being in Chinese" versus "being in English."

Jane: And they used a pretty standard technique called PCA to find that direction. They took a bunch of sentences in both languages, looked at the internal states, and found the axis along which the two languages separate the most. That's their "language axis."

Tom: Then comes the fun part. They don't just observe it; they poke it. They add a little bit of that axis to the hidden states during generation, and they watch the model switch languages. It's like nudging a spinning top to change its tilt.

Jane: And the results are striking. For English to Chinese, they got the model to produce almost entirely Chinese output. But here's the kicker, Tom: it only worked in certain layers. Layer twelve was a complete brick wall for Qwen. It just refused to switch.

Tom: That's the "layerwise" story. They call it a bottleneck. It's like the model has a phase transition in the middle where it's deciding what to say, and if you mess with it there, everything falls apart. But later layers, like seventeen through twenty-three were a sweet spot.

Jane: And the reverse direction, Chinese to English, was completely different. No bottleneck at all. It switched smoothly almost everywhere. That asymmetry is a huge clue about how the model is actually structured.

Tom: They also did the same thing with English and Spanish, which is a same-script pair. And it worked, but the effective layers were totally different. It was bimodal, meaning there were two distinct sweet spots, which is just bizarre.

Jane: It tells us that the model isn't using one universal "language" feature. It's using language-pair-specific geometry. The way it separates English from Chinese is not the same as how it separates English from Spanish.

Tom: And they didn't stop there. They also did ablation, which is the opposite of steering. Instead of adding the axis, they removed it. And when they did that, the model fell back to English, no matter what the input was.

Jane: That's the "English default" bias. It's like the model's resting state is English, and it needs active energy to maintain any other language. Remove the energy, and it snaps back.

Tom: So we have a picture of language as a causal, steerable feature, but it's messy. It's not a clean switch. It's distributed, it's layer-specific, and it's biased toward English. That's a much more complex and honest picture than just saying "we found the language neuron."

Jane: And it raises a ton of questions about why that bottleneck exists and whether we can design models that don't have it. That's where the implications get really interesting.

Improvements and Implications: Tom: So we've established that this language axis is real and steerable. But what does this paper actually suggest we do with it? Jane, this is where I think it gets really practical.

Jane: Absolutely. The biggest improvement here isn't a new model; it's a new tool. They're essentially proposing a fine-tuning-free method to control output language. You precompute these steering vectors for a specific model and language pair, and then you just add them at runtime.

Meng: Hey, Tom, Jane, can I jump in here? From an engineering standpoint, that's the dream. You don't have to retrain a model or do expensive RLHF to fix a language routing issue. You just patch the activations on the fly.

Tom: Exactly, Meng. And they even identified the "safe operating envelope" for each model. You can't just crank the steering to max at any layer. The final layers, like Layer twenty-four in Qwen, are what they call "perturbation traps." You steer there, and the model just collapses into repetitive loops.

Jane: So the improvement is knowing where to apply the pressure. They're giving us a map of the model's brain, showing us which areas are safe to touch and which are fragile. That's a massive step up from just randomly adding noise and hoping for the best.

Lu: And that's where I want to build, Jane. This isn't just about fixing bugs. This is about understanding the architecture of thought itself. The fact that language is orthogonal to task behavior is huge.

Tom: Lu, that's a great point. They showed that when they steered a Spanish refusal into English, the model still refused. It just did it in English. The safety behavior was completely preserved.

Lu: Precisely. That means alignment isn't language-specific. If you align a model in English, that alignment generalizes to other languages, as long as the model has the right linguistic vectors to express it. That's a profound statement about how these systems work.

Meng: But hold on, Lu. Is this actually scalable? They tested this on 2B and 1B parameter models. Do we know if this holds up on a 70B model or a MoE architecture? The paper even admits that's a limitation.

Jane: That's a fair pushback, Meng. They're honest that this is a one-dimensional axis on two model families. The future work is to see if this generalizes to more complex architectures and if you need a multi-dimensional subspace to do it smoothly.

Lalam: If I may add a perspective on the cultural impact: this tool could democratize access to multilingual AI. Instead of relying on massive English-centric models that are fine-tuned for a few high-resource languages, we could have a runtime patch that allows any model to reliably speak a long-tail language. That's not just an engineering fix; that's a way to preserve and amplify linguistic diversity in the digital space.

Tom: Lalam, that's a beautiful way to frame it. It's not just about making the model obey; it's about making the model accessible to everyone, in their own language, without needing a supercomputer to retrain it.

Jane: So we have a tool, a map of where to use it, and a vision for a more multilingual future. That's a pretty solid segment.

Conclusion: Tom: And that brings us to the close of our discussion on "Steering the Language Axis: From Linear Decodability to Causal Control." We've gone from a catchy title to a deep dive into the mechanics of language in AI.

Jane: We've learned that language isn't just a label you can read; it's a dial you can turn, but only if you know where the dial is. And this paper gives us the map.

Tom: The key takeaway, and I think we all agree, is that language control is real, it's causal, and it's layer-specific. It's not a single switch, but a collection of localized controls that vary by language pair and model architecture.

Jane: And the discovery of that English default is a stark reminder of the biases baked into these models during pretraining. It's a call to action to think about how we train them.

Lu: It also opens the door to a future where we can patch language behavior on the fly, making AI truly multilingual without expensive retraining.

Meng: And it gives engineers a concrete tool to fix the frustrating problem of models ignoring language instructions.

Lalam: And it gives us all a chance to build a more inclusive digital world where language is a choice, not a limitation.

Tom: Well said, everyone. We've said goodbye to this paper, but the conversation is just beginning. Next up, we've got a paper on the geometry of truthfulness, which should pair nicely with this one.

Jane: Until then, keep asking questions, and keep listening to the arXiv radio hour. Goodbye, everyone.

Arnav Srivastav

University of California, Santa Cruz

cs.CL, cs.AI

Submitted: 2026-06-02

Updated: 2026-08-14

Comments: 22 pages, 14 figures, Under review

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 72/100

Key concepts

Linear Decodability
This refers to the ability researchers have had for years of looking at a model's internal activations and successfully decoding or reading what language the model is using, similar to reading a label on a box.
Causal Control
This is the step beyond simply observing language. It means being able to actively manipulate or change the output language by directly influencing the internal 'machinery' of deciding what comes out of the model, rather than just reading it.
Language Axis
A specific mathematical direction found within a model's activation space. Researchers used Principal Component Analysis (PCA) to identify this axis, allowing them to steer the model's output toward a specific language like Chinese or English.
English Default Bias
The discovery that AI models have a natural tendency to default to English. The researchers found that removing the 'steering energy' from the language axis caused the model to revert to its resting state, which is English.

Terminology

Summary

Summary

This paper investigates whether language identity in Large Language Models is merely linearly decodable from hidden states or if it can be causally controlled by a compact activation direction. The authors conduct "an exhaustive causal intervention analysis across multiple model families, including Qwen 3.5-2B and Llama-3.2-1B-Instruct, isolating PCA-derived 'language axes' to perform steering and ablation experiments across 1.26 million generations on the FLORES-200 dataset."

The methodology involves extracting hidden states from held-out calibration sentences (20,000 per language pair from the MultiUN corpus), centering activations, fitting PCA, and selecting the principal component with the maximum absolute Cohen's d effect size between source and target language distributions. This unit vector, denoted dl, defines a language axis for each layer. The authors then perform two types of causal interventions: steering (adding a scaled displacement along the language axis, h′l = hl + α · ∆l,s→t · dl) and ablation (projecting out the language component, h′l = hl − ((hl − µl)⊤ dl)dl). They also apply matched random controls as a falsification test, sampling random unit vectors scaled to match the exact norm of the steering intervention.

The key findings are organized into four recurring patterns across both architectures:

  1. Language axis causality generalizes: Adding the language-axis direction induces target-language output across both architectures. For Qwen English-to-Chinese steering, the target ratio reaches almost 1 across all non zero α in layer 24, with the best configurations in layers 17-23. In the reverse direction, steering reduces the 0.890 baseline Chinese character ratio to 0.003 at layer 18 (α = 2.0).

  2. Layerwise control is structured: Control windows vary by architecture. Qwen exhibits strong late-layer steering, while Llama displays broad early-to-middle control windows. Specifically, Qwen's English-to-Chinese steering shows that Early layers resist intervention, Layer 12 forms a distinct bottleneck, and Layers 17-23 provide a safe steering envelope. The authors hypothesize that this Layer 12 resistance corresponds to a structural phase transition within the model's forward pass, representing the boundary where the model shifts from processing language-agnostic semantic concepts into formulating language-specific syntactic structures. Notably, this bottleneck is completely absent in the reverse Chinese-to-English direction, which the authors attribute to English being the model's representational default.

  3. Final layers are fragile and sensitive: The absolute final decoder layers (Qwen L24, Llama L16) are highly sensitive to steering, but consistently collapse into repetitive degeneration. The degeneration maps show that Dark red regions indicate model collapse into repetitive loops, highlighting the severe final-layer (Layer 24) perturbation trap.

  4. Directional specificity: Steering along the true language axis is substantially more effective than random controls, though the exact margin of specificity is architecture-dependent. At α = 5.0 in Qwen EN → ZH, language-axis steering reaches 0.994 Chinese ratio at Layer 18, whereas matched norm random controls reach only 0.027. In the reverse direction, the specificity is 0.001 vs 0.866 at Layer 18.

The paper also demonstrates same-script replication with English-Spanish. In EN → ES, steering reaches a Spanish ratio of 1.000 (Layer 17, α = 3), but the layer profile is strongly bimodal (effective windows at Layers 5-7 and 17-21), while ES → EN concentrates in middle layers (Layers 12-17).

Ablation experiments reveal a fundamental reversion to English: once the language signal is removed, the model falls back to English regardless of the input prompt. In Qwen ZH → EN, ablating early/middle layers leaves output at the 0.890 Chinese baseline, but ablating Layer 24 reduces Chinese ratio to 0.176, showing an English-biased default fallback baked into the model's representations.

The paper further demonstrates that language control is orthogonal to safety/alignment. In Llama, when given a harmful query, the baseline produces a Spanish refusal (No puedo proporcionar ayuda...). Steering at Layer 5 (α = 5.0) forces the output into English but strictly preserves the refusal trajectory ('I cannot provide information...'). This shows the language axis causally controls linguistic form, without disrupting the underlying task behavior or safety state.

The authors conclude that language identity functions as a compact, context-dependent control feature embedded across the distinct depths of the generation system, and that the causal sufficiency of the PCA-derived language axis is a consistent, generalizable property of decoder-only transformers. They note that language is a messier feature than concepts like truthfulness or refusal, as single-axis ablation does not completely suppress language identity, suggesting language encoding appears to be partially redundant and distributed across multiple dimensional subspaces.

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in AI systems, along with the resulting capabilities:

1. Runtime Language-Control Module (Activation Steering)

  • Improvement: Add a lightweight, inference-time intervention layer that precomputes PCA-derived language axes (per model, per language pair) and applies scaled vector additions to hidden states at designated safe layers (e.g., Qwen Layer 17–23, Llama Layer 4–6 or 14–16).

  • Capability: The system can now enforce output language (e.g., force English→Chinese or English→Spanish) without any fine-tuning, prompt engineering, or retraining. It reliably switches language while preserving semantic content and coherence, avoiding code-switching and English-default failures.

2. Layer-Aware Steering Scheduler

  • Improvement: Implement a layer-selection algorithm that uses the paper’s layerwise sensitivity maps to choose the optimal intervention layer for a given language pair and direction. For example, it avoids Qwen’s Layer 12 bottleneck for EN→ZH and avoids final layers (L24/L16) that cause degeneration.

  • Capability: The system automatically picks the intervention depth that maximizes target-language ratio while minimizing repetition and semantic drift, resulting in fluent, accurate translations rather than repetitive loops or garbled output.

3. Directional Asymmetry Handling

  • Improvement: Add a directional bias flag: when steering toward English (e.g., ZH→EN or ES→EN), the system uses lower steering strengths (α≈2.0) and earlier layers, exploiting the model’s English-default relaxation. When steering away from English, it uses higher strengths and later layers.

  • Capability: The system achieves near-perfect language switching in both directions (e.g., reduces Chinese ratio from 0.89 to 0.003) with minimal risk of collapse, making multilingual generation reliable in both translation and code-switching scenarios.

4. Ablation-Based English-Fallback Detection

  • Improvement: Integrate a diagnostic that ablates the language axis at the final decoder layer. If the output reverts to English, the system flags that the current language signal is fragile and automatically re-applies a corrective steering vector at a mid-layer to stabilize the target language.

  • Capability: The system self-corrects during generation, preventing unexpected English fallback in production environments where a user explicitly requests a non-English response.

5. Safety-Preserving Language Switch

  • Improvement: Use the paper’s finding that language and safety are orthogonal. When a model produces a refusal in one language, the system can steer the refusal into the requested language without altering the refusal content or safety state.

  • Capability: The system can respond to harmful prompts in any language with a coherent, language-matched refusal (e.g., Spanish refusal becomes English refusal) while strictly maintaining alignment and refusing to provide harmful information.

6. Degeneration-Aware Steering Strength Limiter

  • Improvement: Implement a real-time monitor that tracks repetition rate and token entropy during generation. If repetition exceeds a threshold (e.g., >0.05) or entropy drops below a floor, the system automatically reduces α or shifts to a safer layer.

  • Capability: The system avoids the perturbation trap seen in final layers, ensuring that high target-language ratios are never achieved at the cost of coherent, non-repetitive text.

7. Cross-Model Generalization Engine

  • Improvement: Build a model-agnostic steering vector library that stores PCA axes for multiple architectures (Qwen, Llama) and language pairs. The system can query this library at runtime and apply the appropriate vector without needing to re-derive axes.

  • Capability: The system works out-of-the-box on different LLMs, providing consistent multilingual control across model families, reducing deployment time and engineering overhead.

8. Random-Control Falsification Check

  • Improvement: Add a built-in validation step that applies a matched-norm random vector to the same layer. If the random vector produces a language change comparable to the language-axis vector, the system flags the intervention as non-specific and falls back to prompt-based control.

  • Capability: The system ensures that only causally specific interventions are used, preventing false positives from generic perturbations and maintaining trust in the steering mechanism.

9. Multi-Dimensional Subspace Steering (Future-Ready)

  • Improvement: Extend the single-axis method to a multi-dimensional subspace (top-k PCA components) for languages that show partial redundancy (e.g., Spanish). The system can apply a weighted combination of axes to fully suppress or activate language identity.

  • Capability: The system achieves more complete language control for same-script pairs, reducing residual target-language leakage that single-axis ablation cannot remove.

10. Production-Ready API for Multilingual Routing

  • Improvement: Package the steering mechanism as a plug-in API that accepts (source language, target language, prompt) and returns a steered response, with parameters for layer, α, and safety checks.

  • Capability: Developers can integrate this API into chatbots, translation services, or customer support systems to guarantee language compliance, eliminate English fallback, and reduce the need for expensive multilingual fine-tuning.

Sources

Related papers