2608.07208-Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes".

Jane: The paper was written by Luc Hazenoot, Zhaochun Ren and Amirhossein Zohrehvand from Leiden Institute of Advanced Computer Science and Leiden University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title and Authors: Tom: So today's paper comes from the Leiden Institute of Advanced Computer Science, written by Luc Hazenoot, Zhaochun Ren, and Amirhossein Zohrehvand, and the title alone tells you it's ambitious — maybe even a little dense.

Jane: Dense how? What's the paper actually claiming in plain language?

Tom: If you're reading a corporate sustainability report and you want to know how much of it is really about the environment, the usual approach is to count words like "green" or "carbon" and hope those words track the concept.

Lu: And hope is the operative word there?

Tom: Exactly. A sentence containing "green" could be about a traffic light — that's literally an example from the paper. So the authors want to measure the judgment a reader forms about the text, not just the vocabulary on the page, and they want to pull that judgment from the internal activations of a large language model.

Jane: Internal activations being the numbers the model computes inside its layers before it produces any output. So the authors are betting those numbers carry more than the final answer the model types out.

Tom: And there's a growing body of evidence that they're right to bet that way. But they test it in a specific domain, which is where ESG comes in.

Lu: I find it fitting that they test on ESG, because that's a domain where the measurement problem is genuinely costly. Banks, regulators, and investors need to know how much of a company's communication concerns social issues or governance structures, and getting that wrong has real consequences.

Meng: And the data itself is expensive to produce. The dataset they use is human-annotated, with three expert annotators per sentence following a shared rubric. Any method that avoids fine-tuning a model on that kind of data is valuable to smaller firms and research groups.

Lalam: That's the bigger picture. If you can read a concept directly from a frozen, out-of-the-box model without fine-tuning, you change the economics of measurement for a whole class of research questions in finance and the social sciences. The authors sit in a computer science institute, but the reach goes well beyond it.

Tom: Right, and they're explicit about the gap they're trying to close. A recent survey covers ten major applications of natural language processing in finance, and none of them monitor the internal activations of language models. This paper brings that idea into an applied benchmark with human labels.

Jane: And they come in with a clear expectation. Based on earlier work, they expect the fancier extraction method, the so-called Recursive Feature Machine, to win.

Tom: The data tells them something else entirely. That's the surprise result, and it's worth getting into the summary now.

Summary: Jane: So we've set up the question and the players, and now we get to what they found. The headline is this — a simple linear probe reading the activations of a frozen model comes within 0 point 6 percentage points of the best fine-tuned classifier on the Environmental pillar, within 1 point 0 on Social, and 2 point 1 on Governance.

Tom: And all of that without any task-specific fine-tuning. A linear probe is just a lightweight classifier trained on the model's internal activations to find a direction that separates "this sentence is about the concept" from "this sentence isn't." The authors train one probe per layer, then combine the per-layer scores through a logistic regression meta-learner.

Lu: "Frozen" means the model's weights never change. The probe is trained on the activations, but the model itself is untouched. That's what keeps the whole thing cheap.

Jane: The crucial comparison is against the model's own answer. They asked each model directly whether a sentence concerned the pillar, and scored the probability it assigned to "yes" versus "no." The probe beat that answer in eleven of twelve model-pillar combinations, by 0 point 043 AUC on average.

Meng: Eleven of twelve — that's decisive.

Tom: It's the proof that the activations carry content the output doesn't report. If reading the internals weren't better than reading the words the model produces, the entire premise of the paper would collapse.

Meng: And the surprise is which method won. They expected the concept vectors from the Recursive Feature Machine to come out on top, based on the recent work that introduced them. Instead the simple probe beat them in eleven of twelve like-for-like comparisons, by 2 point 9 accuracy points on average.

Jane: They were careful about that comparison, though. The probe was evaluated by cross-validation while the concept vectors were evaluated once on a held-out split, so they re-ran the probe on exactly the same training and test sentences. The ordering held.

Lalam: I'd argue that's the most valuable finding for the field. It resets expectations about what the simplest extractor can do. The Recursive Feature Machine still has a role because it produces a continuous score — how strongly a concept is present — instead of just a yes or no. But for plain classification, the probe is the workhorse.

Lu: Though on Social, neither activation-based method beat the embedding baseline. The probe matched it at 0 point 924 against 0 point 925, effectively a tie. So the story isn't that activations crush everything — it's that they're competitive without any fine-tuning, and often better.

Tom: And there's a limitation hiding in plain sight. The continuous score gets reported as distributions, but the paper can't validate its magnitude, because the ESG datasets only carry binary labels. The graded reading is the part that remains unproven.

Jane: Right, and that connects directly to the improvements the paper points toward. Let's talk about those next.

Improvements: Tom: So the results are in, and the probe won. The natural question is what this buys practitioners, and what the paper says still needs work. The main improvement is cost — no domain-specific fine-tuning required, with activations extracted and concept vectors fitted in under four minutes on a single H100 per model and pillar, and the full probe at about five.

Jane: That's a real change for anyone without the data or compute to fine-tune. And the paper makes a clear practical recommendation: the lightweight probe should be the default choice, because it's the strongest method they tested and among the cheapest to run.

Lu: The concept-vector method earns its extra machinery only when the continuous score is itself the object of interest. That's when you want to rank texts by how strongly they relate to a concept rather than just classify them.

Meng: And there's a fascinating finding about wrappers, the template each sentence is wrapped in before going into the model. Holding the model, dataset, and method fixed, changing the wrapper moved accuracy across a 9 point 4-point range. Simply ending it with "Final answer" instead of "Answer" shifted results by 2 point 6 points.

Tom: That's the kind of detail that usually gets buried in a footnote. The paper is honest that wrapper selection is the least controlled step in the pipeline, and their search covered only six wrappers. So we don't actually know how much better either method could do with a truly good wrapper.

Jane: They also give guidance on where the signal lives. For easier concepts, a single well-chosen intermediate layer does almost all the work — the full stack gains under a point of AUC on Environmental. On the harder Governance task the stack earns up to five points, which suggests stacking pays off only when no single direction cleanly separates the classes.

Lu: The future work section is refreshingly concrete. They want ordinal labels to validate the continuous score, a more principled way to design wrappers, and they suggest larger models or financially pre-trained models would carry stronger directions for these pillars.

Meng: There's a smart hypothesis about why model size matters here. Smaller models have likely seen limited ESG text during training, which may be why Environmental sentences score higher than Governance ones across all methods — environmental topics appear far more often in the general corpus.

Lalam: And I appreciate their transparency about what didn't run. They couldn't fit the linear probe on their biggest model, Gemma-4-31b, because the accelerator ran out of memory. On a larger card, they say, it lands at or just below the best Qwen configuration, but you won't find it in the tables.

Tom: That honesty makes the rest of the results easier to trust. They also flag that the same wrapper doesn't transfer across models — one wrapper performs better on Llama while another performs better on Qwen. All of this ties back to how the paper frames the problem on its very first page.

Jane: Let's go back to that opening and look at how they set up the argument.

First Page: Jane: So we've covered the results and the recommendations, and now the first page. What's striking there is how the authors position existing measures. Dictionary methods count concept words, topic models allocate words to topics, and embedding methods compare texts to a concept direction in embedding space — they all score the words a text uses, not the judgment a reader forms about it.

Tom: And that gap shows wherever a concept resists enumeration. How much of an earnings call concerns risk? How much of a report concerns sustainability? You can't write a dictionary that captures that, because the concept lives in the reading, not in the vocabulary.

Lu: The same page cites recent work showing that LLMs encode more internally than they express in their responses. There's even a finding that only a small subspace of activations, concentrated in the intermediate layers, carries what a model can report verbally — the rest drives processing that never surfaces in the output.

Meng: So the paper's contribution is really two things. First, a cheap, off-the-shelf measure of concept content operating on frozen models. Second, a head-to-head comparison showing the simplest extractor is currently the strongest. Prior work never put the two extractors against each other.

Jane: And the dataset gives the test real weight. Two thousand sentences per pillar, drawn from a corpus of 13 point 8 million sentences from annual reports, sustainability reports, and news articles. The annotation design is clever — a thousand sentences are shared across all three pillars, so a single sentence can be positive for more than one pillar at once.

Tom: The authors also stress that keyword filtering doesn't guarantee a true positive. A sentence containing "green" could be about a traffic light rather than the environment. That's precisely why surface measures fall short, and why the human-annotated labels are the benchmark.

Lalam: What I find striking on that first page is the positioning. The finance NLP survey they cite covers ten major applications, none of which monitor internal activations. So the paper does more than offer a new method — it opens up a layer of information that applied work has simply not been using.

Tom: The first page also previews the two design choices every activation-based method faces: how to turn labelled activations into a concept direction, and how to reduce the many token vectors a sentence produces into one fixed-size vector. The experiments vary both.

Jane: And the pooling choice turns out to matter in a messy way. Last-token pooling wins on Environmental, max pooling wins on Social and Governance. There's no universal answer, and the paper says so plainly.

Meng: One of the most human moments in the paper comes later in the discussion — some confident errors sit on sentences whose labels are genuinely arguable. There's an example about a company committed to building a sustainable business that is economically, socially, and environmentally conscious, which the model scores strongly positive but the dataset labels negative. The first page's premise, that surface and judgment diverge, lives right there in the data.

Tom: And the authors don't claim to know who's right. They say a new team of domain experts would need to re-annotate those sentences. That's the scientific temper of the paper, and a good note to carry into our final summary.

Conclusion: Tom: So here's where we land. The paper asks whether the internal activations of a frozen LLM can measure concept content without task-specific fine-tuning, and the answer is yes. The linear probe gets within 0 point 6 to 2 point 1 accuracy points of fine-tuned domain classifiers across the three pillars, and it needs only a few minutes of GPU time per pillar.

Jane: Two control comparisons make that result convincing. The probe beats the same model's own yes-or-no answer in eleven of twelve cases, by an average of 0 point 043 AUC, so the activations genuinely carry content the output doesn't report. And the probe beats the concept-vector method eleven of twelve times when both see identical sentences, so the ordering reflects the extraction method, not the evaluation protocol.

Lu: The continuous score from the concept vectors remains the open thread. It's meant to reflect how strongly a concept is present, but current ESG datasets only carry binary labels. Validating that graded reading has to wait for ordinal annotations.

Meng: And the wrapper problem is the other big loose end. A 9 point 4-point swing from wrapper changes is enormous, and wrapper design remains the least controlled step in the pipeline. Whoever finds a principled way to build wrappers will likely push these numbers higher still.

Lalam: The broader implication is what stays with me. As corporate communication becomes increasingly machine-generated, the gap between what a text appears to say and what it actually carries will only grow. A measure that reads concept content from internal representations rather than surface cues gives us an independent check on that.

Tom: And that's why this paper matters beyond ESG. The method is domain-agnostic — ESG just happens to be the well-annotated test bed. Any research question that asks how much a text concerns a concept can borrow this recipe.

Jane: We'll be watching for the follow-up with ordinal labels and a systematic wrapper study. For now, the message is simple: if you're measuring concepts in text, don't just read the words, and don't just read the model's answer. Read what the model registers inside.

Tom: And with that, we're closing the book on this one. Thanks to everyone listening, and we'll be back soon with the next paper.

Jane: Take care, everyone.