The Signs Were Always There: Training-Free Detection and Control in Raw Transformer Dimensions

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "The Signs Were Always There".

Jane: The gist The standard basis of transformer hidden states is already a trainingfree, architecture-general feature basis for detecting concepts and, in language models, for steering them.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper today, "The Signs Were Always There: Training-Free Detection and Control in Raw Transformer Dimensions." It sounds like they're saying you don't need to train anything to find concepts and control language models using the standard way the transformer works.

Jane: That’s what it sounds like. The authors are suggesting that the hidden states themselves, before any training, already have a structure that lets us read things without learning anything new about the model. It’s really interesting because it bypasses all those traditional methods of representation learning.

Lu: Exactly. They are using something called the Bag of Dims framework where each dimension is treated like a simple binary register, and its sign tells you if it's on or off for a specific concept, and its magnitude tells you how strong that presence is.

Meng: So instead of learning a complex vector space from scratch, we are just looking at the existing numbers in the hidden state output. That sounds like it should be much cheaper to run in production, right?

Tom: It’s not about being cheaper to run, Meng; it’s about finding a way to do something entirely different—detection and steering—using what’s already there. They show that a feature is just any subset of dimensions where the sign pattern stays consistent across tokens belonging to the same category.

Jane: And they discovered how to find these features without any prior knowledge of what concepts exist in the data. They use something called Per-dimension AUC, which means they test how well each individual dimension can separate members of a category from everything else in the vocabulary.

Lu: They do this by taking anchor tokens for a category and then measuring that separation power across all two thousand five hundred sixty dimensions for different models like Qwen and Mistral <ref:2606.12629#pg2>. It seems to work across language, vision, and audio because these individual dimension signs are consistent everywhere.

Meng: But what does this actually mean for deployment? If we have to calculate sign agreements over all dimensions just to identify a concept, that sounds computationally heavy at inference time.

Tom: The paper says that the process of finding the features is essentially unsupervised and scales well. They claim they can discover about one thousand five hundred features per model with a one hundred percent yield using this single-token type cache as their starting point.

Jane: That’s a big number for discovery, Tom. It suggests we could potentially uncover a lot of concepts just by looking at the raw model output, without needing any manual labeling or training sets for that discovery phase.

Lu: And they also proved something about the structure itself—the paper shows that individual dimensions follow their own distinct paths in the hidden state trajectory, and only when you overlay all two thousand five hundred sixty of them do you see the population-level envelope, which is just a population effect of many oscillating dimensions <ref:2606.12629#pg2,individual dimensions follow their own>.

Title and authors: Meng: So they’re saying that even though we can find features without training, there’s still some underlying structure in how those dimensions evolve during training.

Tom: Right. And this leads into the second big part: controlling the model. They introduce what they call a write target, which is a second way to define concepts based on the sign consensus of the output projection over seed tokens.

Jane: That write target is what lets them steer the model during generation, and it’s injected right through the attention output pathway under closed-loop control. That means you can influence what the model writes without ever having to retrain or fine-tune anything for that steering part.

Lu: It's a second catalog that lives on dimensions nearly disjoint from the read catalog, which is the set of dimensions used for detection. They found that this write target is nearly disjoint from the read catalog, with a rank correlation between them of zero point one to one ten percent.

Meng: Disjoint means they aren't using the same dimensions to do both jobs; that’s good for scaling because you don't double up on limited resources. So detection and control are separate skills happening in the same standard basis.

Tom: Precisely, Meng. And one crucial point they make is that reading a concept and controlling it are actually two separate roles within this same basis structure. A read-side sign flip can suppress a concept, but forcing the signs toward an expected direction doesn't necessarily induce it because the high-magnitude read dimensions already have the correct sign.

Jane: That’s a subtle but important distinction for anyone trying to build control systems on top of these models. It suggests that induction needs something separate from detection, which is why they put that write target on dimensions that don't overlap much with the read set.

Lu: I also noticed they looked at the internal structure of the Feed-Forward Networks too—the weights there already show these axis-aligned sign patterns one step earlier than in the residual stream activations.

Meng: So it’s not just about what’s in the main transformer block; it's baked into how the model processes information deeper down, surviving even when you pass those activation signs through a SwiGLU gate.

Tom: And they found that these neuron activations themselves are sign-readable and match the residual stream within a very small margin of plus or minus zero point zero five on per-dimension AUC.

Jane: That’s really cool, Tom. It means the Bag of Dims structure isn't just a neat trick confined to the residual stream; it’s already present in the FFN’s own activation space and persists through that gate mechanism.

Lu: So, we have detection based on sign agreement from tokens, and steering based on a separate sign consensus from the output projection, both operating within this single standard basis structure.

Title and authors: Meng: And for deployment at scale, the paper says the combinatorial capacity of these sign patterns—which is three D minus one features—means there isn't an inherent scaling limit as those models get bigger. That’s good news for us shipping things to production.

Tom: It’s cheap because the only artifact we need to manage is the type cache, and that combinatorial capacity means we can have a lot of distinct features even in huge models. And both methods are training-free—the read catalog from the type cache, and the write target from frozen weights.

Jane: So, to sum up this paper "The Signs Were Always There: Training-Free Detection and Control in Raw Transformer Dimensions," it shows that we can use the standard hidden state basis for both detecting concepts via sign agreement and steering generation via a separate training-free target, without needing any training at all.

Lu: The implication for representation learning is that we don't need to learn complex rotations or learned projections to get these useful semantic features; the structure is already there.

Meng: For production engineering, it means we have two distinct ways to operate on the model—one for identifying what’s there, and one for influencing what comes out—both using only frozen weights and a simple cache artifact.

Tom: So we have a read catalog that detects concepts, and a write target that controls them. Detection and control are separate skills operating on the same substrate.

Jane: It really moves the discussion toward how we can build explicit control mechanisms into these large models without needing massive amounts of labeled data for every single task.

Lu: And they showed this works across seven very different model types, including Qwen three point five-4B, Gemma three-4B, Mistral 7B, and even vision and audio models like DINOv2 and AST <ref:2606.12629#pg1,Qwen 3.5-4B, Gemma 3-4B, Mistral 7B>.

Meng: That cross-modality validation is important; it suggests the underlying physical structure of the transformer training process is what matters, not the specific task we are asking it to perform.

Tom: And that's because the same per-dimension signs appear in every modality, which reflects general transformer training rather than just optimizing for a single language modeling objective.

Jane: So, this paper provides a very clean foundation for using these hidden states as a universal feature basis, regardless of whether you’re looking at text or an image.

Lu: The next step is understanding how to operationalize the steering part reliably in real-world generation pipelines.

Meng: I'm curious about the practical implementation details of that closed-loop control—how stable is that feedback loop when we deploy it?

Tom: We’ll need to see those details, but for now, what this paper gives us is a training-free toolkit for both identifying and influencing concepts in raw transformer dimensions.

The paper's summary: Tom: So, this paper is all about using the standard transformer hidden states—the raw numbers from inside the model—as if they are already a map for understanding concepts and controlling them without any new training data.

Jane: It's like saying you don't need to teach a child what a 'cat' looks like; you just look at how their existing brain structure naturally organizes things, and you can read that organization.

Lu: The core idea is the Bag of Dims framework, where each dimension in the hidden state output is treated like a little binary switch.

Meng: So if we treat those dimensions as on or off signals for different concepts based just on their sign—positive or negative—we can find features without any fancy learning setup.

Tom: Exactly. We have this read catalog, which is built by looking at how consistently the signs of those dimensions stay together across tokens that belong to a certain category.

Jane: And they show you how to discover these features using just a single-token type cache, meaning no massive datasets or training for feature discovery itself is needed.

Lu: They use something called Per-dimension AUC, which measures how well each individual dimension separates members of a category from the rest of the vocabulary.

Meng: So they test every single dimension on its own to see if it’s good at distinguishing between categories like 'dog' and 'chair'. How many features do we get from that process?

Tom: They found that you can discover about fifteen hundred features per model with a one hundred percent yield, which is pretty solid for unsupervised discovery.

Jane: That’s a lot of potential concepts revealed just by looking at the existing model structure. But the paper doesn't stop at just seeing what’s there.

Lu: The real power comes with the second part: steering generation using a write target, which is essentially another catalog built from the sign consensus of those output projections over some seed tokens.

Tom: This write target is nearly separate from the read catalog, with a rank correlation between them around zero point one to one ten percent.

Jane: They inject this control signal directly into the attention output pathway under closed-loop control, which means we can steer the model into fluent text on about sixty two to ninety two percent of twelve different concepts.

Meng: So we have detection that’s based on reading what’s already happening, and steering that’s a separate catalog derived from the weights themselves, not retraining.

Tom: That's right. And they showed you can causally suppress a concept by flipping the signs in the read catalog, but forcing them toward an expected direction doesn't induce it because the high-magnitude dimensions already have that sign.

Jane: That means detection and control are separate skills operating on the same substrate, which is a really clean way to think about it.

Lu: One structural observation they made was that these same axis-aligned patterns appear earlier, in the feed-forward network weights, both in the projection weights and the neuron activations.

Meng: So it's not just happening at the very last layer; this structure is baked into how information flows through the model deeper down.

Tom: And those neuron activations are sign-readable themselves, matching that residual stream within plus or minus zero point zero five on per-dimension AUC.

Jane: It really validates that the Bag of Dims structure isn't just a superficial trick; it’s a fundamental property of how the model is trained.

Lu: And for anyone looking at this from a representation learning standpoint, they found that because dimensions are pairwise independent, you don't need complex cross-dimensional structures to read anything meaningful.

Meng: That's good for scaling because if each dimension doesn't depend too much on the others, we can manage the computational load.

Tom: And this method scales well because you get combinatorial capacity from those sign patterns, which is three D minus one features, so even huge models don't hit an inherent scaling wall here.

Jane: It means we have a training-free toolkit for both identifying and influencing concepts in raw transformer dimensions without needing to learn new representations.

The paper's improvements: Tom: So, after showing how you can read concepts and control them using these raw hidden states, they are suggesting some ways to make this framework even more practical and robust for different situations.

Jane: It sounds like they're looking at how we can get better results when we have different types of data or models, because right now it seems very specific to one setup.

Lu: They propose focusing on cross-modality validation, which is basically proving that these per-dimension signs work the same way for vision models like DINOv2 and audio models like AST as they do for language.

Meng: So if it works across different types of input, that’s better than just proving it works for one language model on text. That would make deployment much wider.

Tom: They are also exploring how to make the steering process more reliable, focusing on how the write target interacts with the read catalog when we're dealing with models that have different architectures or scales.

Jane: It seems they want to ensure that if you switch from a small model like a Qwen three-4B to a Mistral 7B, the detection and control mechanisms still function predictably <ref:2606.12629#pg1>.

Lu: They are also looking at how to improve the feature discovery process, suggesting that we should consider not just one token cache but perhaps more sophisticated ways to select those anchor tokens for different categories.

Meng: What does that mean for us in terms of engineering? Does it mean we need a more complex pipeline to prepare those input tokens?

Tom: It suggests optimizing the input cache is important, but the core idea remains: you can still discover features without training, and then you use those same principles to steer generation.

Jane: And they are also looking at how to better understand why things work—not just that they work—which is crucial when we're trying to build trustworthy systems.

Lu: They point toward the structural observations in the feed-forward network weights, suggesting we should look even closer at those earlier layers for these sign patterns.

Meng: So they're pushing us to check deeper into the model structure, beyond just looking at the final output layer or residual stream.

Tom: It’s about making sure that this isn't just a local phenomenon in one part of the transformer; it’s a general property of the weights themselves.

Jane: And for those of us who only listen to this show, it means we should keep an eye on how these underlying structural properties are preserved across different model sizes and modalities.

Lu: The authors are also implicitly pointing toward how the causal roles of reading and writing interact, refining exactly when suppression versus induction happens during inference.

Meng: That’s important because if we can't control the exact moment a concept is suppressed or induced, it gets messy in real-time generation.

Tom: So they are refining the rules for when to use that read-side sign flip versus when to rely on the write target for induction.

Jane: It’s about getting a more precise toolkit for intervention, moving beyond just saying "we can do it" to saying "here is exactly how we do it reliably."

Conclusion: Tom: So, to wrap up, this paper, "The Signs Were Always There: Training-Free Detection and Control in Raw Transformer Dimensions," shows we can use standard transformer hidden states for both detecting concepts and steering generation without any training on top of the model.

Jane: It really boils down to using the existing structure of those dimensions to find semantic features, and then using a separate target catalog to guide what the model actually generates.

Lu: The big implication is that you don't need massive labeled datasets just to get useful concept representations; you can pull them directly from the raw weights.

Meng: From an engineering standpoint, it means we have two distinct, training-free ways to operate on a model—one for detection and one for control—which should make deployment much more flexible.

Tom: Yeah, and that's because the combinatorial capacity of those sign patterns means this approach has no inherent scaling limit as models get bigger.

Jane: It’s cheap to use because the only artifact you really need to manage is that small type cache, and both catalogs are derived from frozen weights.

Lu: The structure itself is already there; the axis-aligned patterns show up in the feed-forward network weights even before you get to the residual stream activations.

Meng: So we’re looking deeper into the model architecture to see where all this sign information is actually hiding, not just at the very top layer.

Tom: Exactly. And that's what makes it so robust; reading and controlling are separate skills operating on the same substrate, which is a really neat mechanism for intervention.

Jane: It moves us toward building explicit control mechanisms into these large models without needing massive amounts of labeled data for every single task.

Lu: The next step is understanding how to operationalize that steering part reliably in real-world generation pipelines, especially when dealing with different model architectures.

Meng: I’m curious about the practical implementation details of that closed-loop control—how stable is that feedback loop when we deploy it?

Tom: We’ll need to see those details, but for now, what this paper gives us is a training-free toolkit for both identifying and influencing concepts in raw transformer dimensions.