2608.06969-Finding Usable Weight Mechanisms with Tiled SVD

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Finding Usable Weight Mechanisms with Tiled SVD".

Jane: The paper was written by Ash Manvi and Samreena Tajreen from Aquin Labs.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We've got a fresh one from Aquin Labs — Ash Manvi and Samreena Tajreen — and it's about dragging mechanistic interpretability out of the proxy world. Instead of training sparse autoencoders to name features, they go straight into the weight matrices.

Jane: And I love the phrase they use — "mechanism mounts." Each one is a triple: a trigger, a write, and a strength. The identity is the weight rule itself, not a verbal label from a dictionary.

Lu: So this is SVD-based. Column-tiled SVD, actually — they slice each weight matrix into tiles and factor each tile separately. Whole-matrix SVD spreads everything out; tiling keeps local structure.

Meng: Right, and then they test it hard. They don't just check whether a tile reconstructs itself — that metric is rigged. They measure full-write energy lift on real forwards.

Jane: On Gemma-2-2B, all seven linear maps per layer. Every one of the 182 site-layers passes. Residual writes get the full treatment — chunking, coverage, and a steer-into-the-residual-stream causal check.

Tom: 52 of 52 residual site-layers with A/B/C, 130 of 130 others with A/B. The headline number is genuinely clean: 182 out of 182.

Lalam: And why does this matter? Because interpretability research mostly describes a model through an external codebook. This paper says: look at the weights themselves. The question "what does this direction mean?" gets answered by the weight rule that writes it.

Lu: They're careful not to overclaim, though. No human-readable concept names, and no claim to replace sparse autoencoders for concept discovery.

Jane: They're honest that novelty is thinner — singular vectors of transformers have been looked at before. The wedge is fair chunking, and a measurement stack you can reuse.

Meng: So the deliverable is less "we found features" and more "here's a pre-registered suite that judges mounts honestly." They release code, corpus builder, tests.

Tom: And we're going to walk through it page by page, because the interesting stuff is in the details — how they build the mounts, and how they dodge the tautological metrics.

Jane: Next up: the first page, where they set up the problem. Let's see how they frame the whole thing.

Page 1: Jane: Where we left off: the paper promises a measurement suite that judges weight mechanisms directly, not through a proxy dictionary. Page one is where they motivate that move.

Tom: They open with the standard tool — sparse autoencoders trained on activations, labeled from max-activating text. Those atlases are useful, but the identity lives in the learned dictionary, not in the network weights.

Lu: Right, and that's the crack they're prying at. If interpretation lives in a separately trained codebook, then "what this direction means" gets answered in proxy space. The weight rule that actually writes into the residual stream stays implicit.

Meng: Then they mention prior work — singular vectors of MLP and attention matrices, read through the unembedding, form interpretable token clusters. So SVD as a lens isn't new.

Jane: Exactly. They cite Millidge and Black on SVD being interpretable, then Xue and Andrzejak framing singular modes as detector-effector units. Their point: those used SVD as a lens or circuit primitive, not as a fair test of which chunking of W yields usable on-distribution mechanisms.

Tom: And there's the keyword — "chunking." You have a big weight matrix. How do you slice it so the pieces are actually usable mechanisms?

Lu: Because the answer isn't obvious. A global SVD gives you abstract directions, but a transformer writes through local column blocks. Tiling respects that locality.

Meng: I like that they name the object early — a mount is a triple: trigger, write, strength. The trigger lives in the input columns, the write lands in the output space, and σ says how strong the coupling is.

Jane: And identity is the weight rule, not a phrase like "the king direction." That's a philosophical commitment, honestly. Meaning is in the mechanism.

Tom: They also preview the evaluation — full-write energy lift rather than tile-local lift, because tile-local metrics can be gamed. That's going to be a recurring theme.

Lu: The page ends before the method details, but the thesis is already sharp: don't label the dictionary; find the mounts in the weights and test whether they actually write.

Jane: Which sets up page two beautifully — they introduce the model, the seven linear maps, and which ones get the full causal treatment.

Page 2: Tom: Picking up: we've got the thesis — mounts in the weights, not labels in a proxy. Page two makes it concrete: the model, the sites, the exact linear maps.

Jane: Gemma-2-2B, 26 layers. Each layer exposes seven linear maps. Two of them are residual writes — mlp.down and attn.o. Those get the full suite, A/B/C.

Lu: And the other five — mlp.gate, mlp.up, attn.q, attn.k, attn.v — get A/B only. The reasoning is clean: their outputs aren't residual writes, so aligning to the unembedding isn't the right causal metric.

Meng: The figure on this page marks them in orange and green — ABOnlyLinear versus ResidualWriteLinear. Nice touch: the residual writes are the ones that directly change the residual stream.

Tom: For the corpus, they use WikiText-2 raw train. They collect all tokens — that's 86,109 — then subsample to 16,384 with seed 0 to keep memory in check. Simple, reproducible, pre-registered.

Jane: What I appreciate is the honesty about defaults — site-aware tile sizes. Residual and MLP maps get tile width 512, attn.k and attn.v get 256, attn.q gets 128 with a 64 fallback. Those aren't magic numbers; they're tuned per site.

Lu: It smells like a lot of engineering went into making the test fair. They're not picking one tile size and hoping it works everywhere.

Meng: And the mention of "effective path mounts" already peeks through the text — for mlp.up and attn.v, the raw module weights won't cut it. The map used on-distribution isn't the raw matrix.

Jane: Right — that's coming in a later page, but it's the most interesting design decision in the paper, honestly. They bake the gate activation into the up-projection.

Tom: So page two sets the stage: fixed model, fixed corpus, clear split between residual writes and everything else. The machinery comes next.

Lu: And it's a lot of machinery — tile-SVD mounts, trigger coefficients, the energy lift. That's page three.

Page 3: Jane: So we're on page three, and this is where the math gets real. Column-tiled SVD: slice W's input columns into tiles, factor each tile with SVD.

Tom: Each tile yields modes — the left singular vector becomes the write direction u, the right singular vector is the trigger v, and the singular value σ is the strength. A mount is just that triple with site and layer metadata.

Lu: The clever bit is the trigger coefficient: a_t,j = x_t · v_j. You run a real forward, you get site inputs x, and the trigger coefficient tells you how strongly this mount fires on that token.

Meng: Then the tile write is the corresponding column block transposed. So the trigger coefficient times the write direction reconstructs the tile's contribution. It's a lightweight, linear story about how the weight behaves.

Jane: And they're upfront that SVD identity holds almost always by construction — correlation above 0.99, slope error below 0.05. That's a sanity check, not a proof. It can't separate usable mounts from unused ones.

Tom: Then comes the crucial turn: tile-local energy lift is a trap. They show it favors one-column tiles tautologically — the lift sits around 0.999 for column sampling.

Lu: That's the negative result in miniature. If you measure inside the tile, you'll always win. So they switch to full-write energy lift — measuring against the site's actual write tensor ∆h on real forwards.

Meng: Random directions seeded with a fixed seed — 10007 plus j times 997 — give you the baseline. The mount has to beat random directions in the full residual write, not in its own little tile.

Jane: Four constructions compete under a matched mount budget: per-tile SVD, whole-matrix SVD, high-norm column sampling, and random. That's the arena.

Tom: And it's a fair arena, because the budget is matched. Same number of mounts, same test, and the tile SVD has to earn its lift in the full write.

Lu: The hook for next page: how do they know the coverage isn't just redundancy? That's the saturation analysis — and the causal steer check.

Page 4: Tom: We've got the mounts, we've got the full-write lift. Page four asks: does coverage actually saturate, and can you steer with these things?

Jane: Coverage saturation — they measure what fraction of the weight's Frobenius norm the per-tile reconstruction keeps, then build a sparse dictionary of mount directions. Select top 8 mounts per token, reconstruct by least squares.

Lu: The key metric is coverage lift versus a random dictionary of matched size. Saturation means the lift peaks early — by one or two modes per tile — and doesn't collapse afterward.

Meng: The floors are site-dependent: 0.25 for residual writes, 0.15 for other maps, 0.08 for the effective up and v paths. And the early sweep window has to sit within 0.08 of the peak. If extra modes just add redundancy, you've saturated.

Tom: Then Experiment C — the causal steer. For residual writes only, they read the final-logit geometry of a write direction through the unembedding: approximate final RMSNorm, then t = Wlm u.

Jane: Steering adds α=2 on the post-attention or post-FF RMSNorm module — not on the projection alone. They average last-token logit changes over eight texts and compute Spearman ρ between Δlogits and the unembed readout.

Lu: The pass bar is modest — ρ ≥ 0.05 or top-20 Jaccard ≥ 0.05 — but it's required only for residual writes in layer 6 or above. Early layers report C but don't fail when alignment is weak.

Meng: That depth-conditioned rule is a design choice they flag honestly. The curve justifies it; they'll show the numbers.

Tom: And then the effective-path mounts — my favorite part. Raw mlp.up and attn.v weights fail the residual-shaped tests because the on-distribution map isn't the raw matrix.

Jane: For mlp.up they build W* = diag(ḡ)W_up, using corpus-mean gate activations. For attn.v, they do a ridge least-squares fit from x to mixed-v. You mount the path that actually gets used.

Lu: It's a subtle point. The raw v-projection writes something, but by the time it goes through attention's mixing and the output projection, the effective write is something else. So you extract from the composed map.

Meng: The pass criteria table at the bottom ties it together — A1 through A4, B1, C1 — with explicit margins. A1 needs tile lift above random plus 0.005, A2 within 0.002 of column sampling, and so on.

Tom: Next page, they stop defining the suite and actually run it. We get the first real numbers from Experiment A.

Jane: And I'm curious whether the chunking claim survives contact with the full model.

Page 5: Tom: Page five, and the suite goes live. They build the WikiText-2 corpus, run the full thing: all layers, all sites, on CUDA. 16,384 tokens subsampled from 86,109 collected.

Jane: The headline: all seven sites × 26 layers pass — 182 of 182. Residual A/B/C are 52 of 52, others A/B are 130 of 130. But the per-experiment numbers are where it gets interesting.

Lu: Experiment A is the chunking test. Tile SVD beats whole-matrix SVD, column sampling, and random at every depth — except mlp.down layer 25, where tile and whole are basically tied at 0.013.

Meng: That's the honest outlier. They don't hide it — the table shows layer 25 mlp.down tile at 0.013 and whole at 0.013. A4 still passes on ratio, but the margin is dead flat.

Jane: Meanwhile layer 18 attn.o is the big winner — tile at 0.383 versus whole at 0.102. Column sampling barely registers, 0.011. Random is essentially zero everywhere.

Tom: What's the intuition? A single global SVD spreads local structure across directions that don't align with how the network writes. Tiling respects the column blocks the attention head or MLP neuron actually uses.

Lu: And they're careful that column sampling — which wins the tile-local metric — loses badly on full-write lift. That's the tautology exposed: locally perfect, globally useless.

Meng: The "fewer mounts" detail is nice, too. attn.o has input dimension 2048, so it gets fewer mounts than mlp.down, and it still wins A everywhere. The tiling advantage isn't just about mount count.

Jane: So chunking works for residual writes on real forwards. The next question is coverage — does the write energy saturate quickly, or do you need many modes? That's Experiment B, on page six.

Tom: And given the table on this page, I'd bet saturation kicks in early.

Page 6: Jane: Page six, Experiment B: coverage versus modes per tile. The sweep goes k = 1, 2, 4, 8, 16. Does adding modes keep buying write coverage?

Tom: The answer is no — it saturates almost immediately. Residual writes show high coverage lift at m=1 or m=2, then flat. Extra modes mostly add redundancy.

Lu: Look at layer 0 mlp.down: m=1 gives 0.225, m=2 jumps to 0.279, m=4 stays 0.279, m=8 goes to 0.302, m=16 drops slightly to 0.298. You've basically extracted everything by the second mode.

Meng: Layer 6 attn.o is even more dramatic — 0.779 at m=1, peaks at 0.793 at m=2, then drifts down. The early window rule catches exactly this: the peak is in the first couple modes, and it doesn't collapse.

Jane: So all 52 residual site-layers pass B1, and the saturation rule does real work — it rejects a curve that only climbs with more modes, because that would mean the tiling isn't capturing the write structure.

Tom: Then Experiment C — the depth curve. They inject after post-norm RMSNorm, steer with α=2, and measure Spearman against the unembed readout of the write direction.

Lu: And the curve is beautiful. On mlp.down, early layers sit near zero — roughly −0.03 to 0.07, waived — then onset layers 6-8 climb to 0.52, mid layers 9-12 around 0.15 to 0.54, and late layers 18-24 reach 0.69. Final layer, 0.91.

Meng: attn.o is shallower but still rises — mid layers around 0.53 to 0.55, final layer 0.75. The key point: mid-depth attn.o no longer fails C1. With post-norm injection, the alignment is genuine.

Jane: And the waiver for early layers is motivated by that curve — ρ hovers near zero until roughly layer 6, then takes off. It's not a free pass; it's reading the data.

Tom: The aggregate verdict on page seven pulls all this together: 182 of 182, with the tier structure spelled out.

Lu: I want to see how they frame what they did and didn't contribute — the discussion on novelty.

Page 7: Tom: Page seven lays out the aggregate verdict — the tier table. Residual A/B/C: mlp.down 26 of 26, attn.o 26 of 26. Other A/B only: 130 of 130. Total 182 of 182 GO.

Jane: And then the discussion starts, and I appreciate the restraint. They say the measurement stack is the durable part of the work — not the discovery of features, but the honest way of testing them.

Lu: They're blunt about novelty being thinner. Singular vectors of transformer weights, detector-effector units, unembed readouts — all exist. They cite Millidge and Black, Xue and Andrzejak, the circuit work from last year.

Meng: The wedge is fair chunking, a negative result about tile-local metrics, coverage saturation, and the depth-conditioned causal check. Those four pieces packaged as a reproducible suite.

Jane: And they explicitly don't claim human-readable concept names, and don't claim to replace sparse autoencoders for concept discovery. That's a clean boundary — this is about mechanisms, not concepts.

Tom: The scope stays sharp, too. Experiment C applies only where u is a residual direction. Raw mlp.up and attn.v weights fail residual-shaped A/B by design — the supported object is the effective path.

Lu: Honest scope is rare in this literature. They're telling you exactly which claims are load-bearing and which sites need special handling.

Meng: All numbers are for Gemma-2-2B on a WikiText-2 subsample — single model family, single corpus. That's not a weakness they hide; it's a limitation they name.

Jane: Which brings us to page eight, where the limitations get their full airing. I've got a feeling the hardest part is the meaning question — energy lift is not human meaning.

Tom: And that's the gap between "we can steer it" and "we know what it means."

Page 8: Jane: Page eight is the limitations section, and they don't soften anything. Single model family and size — Gemma-2-2B only. Experiment C only for residual writes. The WikiText-2 subsample may bias which mounts look strong.

Tom: Then the big one: energy lift is not human meaning. Mounts carry no semantic labels. You can have a mount that writes hard, but the paper won't tell you what it "means."

Lu: And the steering protocol is narrow — short texts, fixed α=2, last-token logits after post-sublayer RMSNorm. That's a thin slice of the behavioral space, and they know it.

Meng: The C1 waiver for early layers is flagged as a design choice. They say it's principled from the depth curve, but early ρ should always be reported. Transparency over convenience.

Jane: mlp.down layer 25 is called out as marginal on A4 — tile equals whole — and attn.o runs with fewer mounts than mlp.down. The table on page five already showed us that.

Tom: Raw mlp.up and attn.v module weights fail residual-shaped A/B — that's a strong statement. If you don't build the effective path, you don't get a passing mount.

Lu: I think that's actually a gift to the field. Negative results about naive approaches save people months of dead-end work.

Meng: So the limitations aren't disclaimers for show. They'd rather shrink the claim than inflate it.

Jane: And the conclusion on page nine ties it together — they extract tile-SVD mounts, score them with full-write lift, coverage saturation, and the depth-conditioned steer check. 182 of 182, with effective-path mounts for up and v.

Tom: The release includes library code, corpus builder, experiment entrypoint, unit tests. Reproducibility is the point.

Lu: Let's wrap this up properly — how would we tell someone why this paper matters?

Conclusion: Tom: So we land the plane. The paper gives us a repeatable way to pull mechanisms out of raw weights — tile-SVD mounts — and a test suite that doesn't fool itself.

Jane: Tile beats whole-matrix SVD, column sampling, and random under a matched budget. Coverage saturates by one or two modes. Post-norm steering aligns with the unembed more and more as depth increases. 182 of 182 site-layers pass.

Lu: The biggest thing they're handing the field is a measuring stick that refuses the easy win. Tile-local metrics are out; full-write energy lift is in.

Meng: And the effective-path move — mounting mlp.up through the mean gate, attn.v through ridge least-squares — that's the part I'll steal. Raw weights lie about what the network does.

Jane: They didn't discover new features with human labels. They built the scaffold for finding mechanisms that actually write, and they shipped the code.

Tom: It's a benchmark, a negative result, and a philosophical stance in one package: identity is the weight rule, not a phrase.

Lu: The limits are real — one model, one corpus, no semantic labels. But the protocol generalizes. Run it on another family, another size, and you learn something.

Jane: Goodbye to this paper, then — and a nice handoff, because the next one on our list picks at the same question from the distribution side.

Tom: Exactly — if mounts are the mechanisms, the next question is how they organize across a model's lifetime. Stick around.