Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes

arXiv:2608.07208 · cs.CL, cs.AI, cs.LG, econ.GN, q-fin.EC · Submitted 2026-08-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes".

Jane: The paper was written by Luc Hazenoot, Zhaochun Ren and Amirhossein Zohrehvand from Leiden Institute of Advanced Computer Science and Leiden University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title and Authors: Tom: So today's paper comes from the Leiden Institute of Advanced Computer Science, written by Luc Hazenoot, Zhaochun Ren, and Amirhossein Zohrehvand, and the title alone tells you it's ambitious — maybe even a little dense.

Jane: Dense how? What's the paper actually claiming in plain language?

Tom: If you're reading a corporate sustainability report and you want to know how much of it is really about the environment, the usual approach is to count words like "green" or "carbon" and hope those words track the concept.

Lu: And hope is the operative word there?

Tom: Exactly. A sentence containing "green" could be about a traffic light — that's literally an example from the paper. So the authors want to measure the judgment a reader forms about the text, not just the vocabulary on the page, and they want to pull that judgment from the internal activations of a large language model.

Jane: Internal activations being the numbers the model computes inside its layers before it produces any output. So the authors are betting those numbers carry more than the final answer the model types out.

Tom: And there's a growing body of evidence that they're right to bet that way. But they test it in a specific domain, which is where ESG comes in.

Lu: I find it fitting that they test on ESG, because that's a domain where the measurement problem is genuinely costly. Banks, regulators, and investors need to know how much of a company's communication concerns social issues or governance structures, and getting that wrong has real consequences.

Meng: And the data itself is expensive to produce. The dataset they use is human-annotated, with three expert annotators per sentence following a shared rubric. Any method that avoids fine-tuning a model on that kind of data is valuable to smaller firms and research groups.

Lalam: That's the bigger picture. If you can read a concept directly from a frozen, out-of-the-box model without fine-tuning, you change the economics of measurement for a whole class of research questions in finance and the social sciences. The authors sit in a computer science institute, but the reach goes well beyond it.

Tom: Right, and they're explicit about the gap they're trying to close. A recent survey covers ten major applications of natural language processing in finance, and none of them monitor the internal activations of language models. This paper brings that idea into an applied benchmark with human labels.

Jane: And they come in with a clear expectation. Based on earlier work, they expect the fancier extraction method, the so-called Recursive Feature Machine, to win.

Tom: The data tells them something else entirely. That's the surprise result, and it's worth getting into the summary now.

Summary: Jane: So we've set up the question and the players, and now we get to what they found. The headline is this — a simple linear probe reading the activations of a frozen model comes within 0 point 6 percentage points of the best fine-tuned classifier on the Environmental pillar, within 1 point 0 on Social, and 2 point 1 on Governance.

Tom: And all of that without any task-specific fine-tuning. A linear probe is just a lightweight classifier trained on the model's internal activations to find a direction that separates "this sentence is about the concept" from "this sentence isn't." The authors train one probe per layer, then combine the per-layer scores through a logistic regression meta-learner.

Lu: "Frozen" means the model's weights never change. The probe is trained on the activations, but the model itself is untouched. That's what keeps the whole thing cheap.

Jane: The crucial comparison is against the model's own answer. They asked each model directly whether a sentence concerned the pillar, and scored the probability it assigned to "yes" versus "no." The probe beat that answer in eleven of twelve model-pillar combinations, by 0 point 043 AUC on average.

Meng: Eleven of twelve — that's decisive.

Tom: It's the proof that the activations carry content the output doesn't report. If reading the internals weren't better than reading the words the model produces, the entire premise of the paper would collapse.

Meng: And the surprise is which method won. They expected the concept vectors from the Recursive Feature Machine to come out on top, based on the recent work that introduced them. Instead the simple probe beat them in eleven of twelve like-for-like comparisons, by 2 point 9 accuracy points on average.

Jane: They were careful about that comparison, though. The probe was evaluated by cross-validation while the concept vectors were evaluated once on a held-out split, so they re-ran the probe on exactly the same training and test sentences. The ordering held.

Lalam: I'd argue that's the most valuable finding for the field. It resets expectations about what the simplest extractor can do. The Recursive Feature Machine still has a role because it produces a continuous score — how strongly a concept is present — instead of just a yes or no. But for plain classification, the probe is the workhorse.

Lu: Though on Social, neither activation-based method beat the embedding baseline. The probe matched it at 0 point 924 against 0 point 925, effectively a tie. So the story isn't that activations crush everything — it's that they're competitive without any fine-tuning, and often better.

Tom: And there's a limitation hiding in plain sight. The continuous score gets reported as distributions, but the paper can't validate its magnitude, because the ESG datasets only carry binary labels. The graded reading is the part that remains unproven.

Jane: Right, and that connects directly to the improvements the paper points toward. Let's talk about those next.

Improvements: Tom: So the results are in, and the probe won. The natural question is what this buys practitioners, and what the paper says still needs work. The main improvement is cost — no domain-specific fine-tuning required, with activations extracted and concept vectors fitted in under four minutes on a single H100 per model and pillar, and the full probe at about five.

Jane: That's a real change for anyone without the data or compute to fine-tune. And the paper makes a clear practical recommendation: the lightweight probe should be the default choice, because it's the strongest method they tested and among the cheapest to run.

Lu: The concept-vector method earns its extra machinery only when the continuous score is itself the object of interest. That's when you want to rank texts by how strongly they relate to a concept rather than just classify them.

Meng: And there's a fascinating finding about wrappers, the template each sentence is wrapped in before going into the model. Holding the model, dataset, and method fixed, changing the wrapper moved accuracy across a 9 point 4-point range. Simply ending it with "Final answer" instead of "Answer" shifted results by 2 point 6 points.

Tom: That's the kind of detail that usually gets buried in a footnote. The paper is honest that wrapper selection is the least controlled step in the pipeline, and their search covered only six wrappers. So we don't actually know how much better either method could do with a truly good wrapper.

Jane: They also give guidance on where the signal lives. For easier concepts, a single well-chosen intermediate layer does almost all the work — the full stack gains under a point of AUC on Environmental. On the harder Governance task the stack earns up to five points, which suggests stacking pays off only when no single direction cleanly separates the classes.

Lu: The future work section is refreshingly concrete. They want ordinal labels to validate the continuous score, a more principled way to design wrappers, and they suggest larger models or financially pre-trained models would carry stronger directions for these pillars.

Meng: There's a smart hypothesis about why model size matters here. Smaller models have likely seen limited ESG text during training, which may be why Environmental sentences score higher than Governance ones across all methods — environmental topics appear far more often in the general corpus.

Lalam: And I appreciate their transparency about what didn't run. They couldn't fit the linear probe on their biggest model, Gemma-4-31b, because the accelerator ran out of memory. On a larger card, they say, it lands at or just below the best Qwen configuration, but you won't find it in the tables.

Tom: That honesty makes the rest of the results easier to trust. They also flag that the same wrapper doesn't transfer across models — one wrapper performs better on Llama while another performs better on Qwen. All of this ties back to how the paper frames the problem on its very first page.

Jane: Let's go back to that opening and look at how they set up the argument.

First Page: Jane: So we've covered the results and the recommendations, and now the first page. What's striking there is how the authors position existing measures. Dictionary methods count concept words, topic models allocate words to topics, and embedding methods compare texts to a concept direction in embedding space — they all score the words a text uses, not the judgment a reader forms about it.

Tom: And that gap shows wherever a concept resists enumeration. How much of an earnings call concerns risk? How much of a report concerns sustainability? You can't write a dictionary that captures that, because the concept lives in the reading, not in the vocabulary.

Lu: The same page cites recent work showing that LLMs encode more internally than they express in their responses. There's even a finding that only a small subspace of activations, concentrated in the intermediate layers, carries what a model can report verbally — the rest drives processing that never surfaces in the output.

Meng: So the paper's contribution is really two things. First, a cheap, off-the-shelf measure of concept content operating on frozen models. Second, a head-to-head comparison showing the simplest extractor is currently the strongest. Prior work never put the two extractors against each other.

Jane: And the dataset gives the test real weight. Two thousand sentences per pillar, drawn from a corpus of 13 point 8 million sentences from annual reports, sustainability reports, and news articles. The annotation design is clever — a thousand sentences are shared across all three pillars, so a single sentence can be positive for more than one pillar at once.

Tom: The authors also stress that keyword filtering doesn't guarantee a true positive. A sentence containing "green" could be about a traffic light rather than the environment. That's precisely why surface measures fall short, and why the human-annotated labels are the benchmark.

Lalam: What I find striking on that first page is the positioning. The finance NLP survey they cite covers ten major applications, none of which monitor internal activations. So the paper does more than offer a new method — it opens up a layer of information that applied work has simply not been using.

Tom: The first page also previews the two design choices every activation-based method faces: how to turn labelled activations into a concept direction, and how to reduce the many token vectors a sentence produces into one fixed-size vector. The experiments vary both.

Jane: And the pooling choice turns out to matter in a messy way. Last-token pooling wins on Environmental, max pooling wins on Social and Governance. There's no universal answer, and the paper says so plainly.

Meng: One of the most human moments in the paper comes later in the discussion — some confident errors sit on sentences whose labels are genuinely arguable. There's an example about a company committed to building a sustainable business that is economically, socially, and environmentally conscious, which the model scores strongly positive but the dataset labels negative. The first page's premise, that surface and judgment diverge, lives right there in the data.

Tom: And the authors don't claim to know who's right. They say a new team of domain experts would need to re-annotate those sentences. That's the scientific temper of the paper, and a good note to carry into our final summary.

Conclusion: Tom: So here's where we land. The paper asks whether the internal activations of a frozen LLM can measure concept content without task-specific fine-tuning, and the answer is yes. The linear probe gets within 0 point 6 to 2 point 1 accuracy points of fine-tuned domain classifiers across the three pillars, and it needs only a few minutes of GPU time per pillar.

Jane: Two control comparisons make that result convincing. The probe beats the same model's own yes-or-no answer in eleven of twelve cases, by an average of 0 point 043 AUC, so the activations genuinely carry content the output doesn't report. And the probe beats the concept-vector method eleven of twelve times when both see identical sentences, so the ordering reflects the extraction method, not the evaluation protocol.

Lu: The continuous score from the concept vectors remains the open thread. It's meant to reflect how strongly a concept is present, but current ESG datasets only carry binary labels. Validating that graded reading has to wait for ordinal annotations.

Meng: And the wrapper problem is the other big loose end. A 9 point 4-point swing from wrapper changes is enormous, and wrapper design remains the least controlled step in the pipeline. Whoever finds a principled way to build wrappers will likely push these numbers higher still.

Lalam: The broader implication is what stays with me. As corporate communication becomes increasingly machine-generated, the gap between what a text appears to say and what it actually carries will only grow. A measure that reads concept content from internal representations rather than surface cues gives us an independent check on that.

Tom: And that's why this paper matters beyond ESG. The method is domain-agnostic — ESG just happens to be the well-annotated test bed. Any research question that asks how much a text concerns a concept can borrow this recipe.

Jane: We'll be watching for the follow-up with ordinal labels and a systematic wrapper study. For now, the message is simple: if you're measuring concepts in text, don't just read the words, and don't just read the model's answer. Read what the model registers inside.

Tom: And with that, we're closing the book on this one. Thanks to everyone listening, and we'll be back soon with the next paper.

Jane: Take care, everyone.

Luc Hazenoot, Zhaochun Ren, Amirhossein Zohrehvand

Leiden Institute of Advanced Computer Science · Leiden University

cs.CL, cs.AI, cs.LG, econ.GN, q-fin.EC

Submitted: 2026-08-07

Updated: 2026-08-10

Comments: 19 pages, 1 figure, 7 tables

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 55/100

The gist: Measured Concept Content from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes The paper asks whether the internal activations of a frozen Large Language Model (LLM) can measure

Key concepts

Linear probe
A lightweight classifier trained on the internal activations of a frozen language model to find a direction that separates sentences about a concept from those that aren't. It doesn't change the model's weights, making it cheap and fast to train.
Concept vectors
Directions in the model's activation space that represent a concept, extracted using methods like the Recursive Feature Machine. They produce a continuous score for how strongly a concept is present, unlike the binary output of a probe.
Frozen model
A language model whose weights are not updated during training. Only the probe or concept vectors are trained on the model's activations, keeping the process cheap and avoiding task-specific fine-tuning.
Wrapper
The template or prompt surrounding a sentence before it's fed into the model. The episode notes that changing the wrapper can swing accuracy by up to 9.4 points, making it a critical but poorly controlled step.

Terminology

Summary

Measured Concept Content from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes

The paper asks whether the internal activations of a frozen Large Language Model (LLM) can measure concept content in text well enough to substitute for task-specific fine-tuning, and which extraction method—linear probing or the Recursive Feature Machine (RFM) concept-vector method—reads those activations best.

The authors argue that existing measures of how much a text is about a concept read only the surface: "Dictionary counts concept words, topic models allocate words to topics, and embedding methods compare texts to a concept direction in embedding space... What these measures share is that they score the words a text uses, not the judgment a reader forms about it. The motivating premise is that LLMs encode more in their internal activations than they express in their responses" (citing Azaria & Mitchell, 2023; Beaglehole, Radhakrishnan, et al., 2026; Burns et al., 2023; Gurnee et al., 2026; Marks & Tegmark, 2024). In applied finance, however, no [major NLP application] involves monitoring the internal activations of LLMs (citing Du et al., 2025). The paper builds on the linear representation hypothesis—concepts are linearly accessible as directions in activation space—and on two prior lines of work: Beaglehole, Radhakrishnan, et al. (2026) extracting concept vectors with the RFM algorithm, and Yang et al. (2024) projecting hidden states onto a learned concept direction to obtain continuous text-based measures. What is missing is a controlled comparison on an applied benchmark.

The authors used an embedding baseline (Qwen-3-embedding-8b, with a logistic regression on the embeddings), and four frozen decoder-only LLMs: Llama-3.1-8b-it (32 layers, 8B parameters), Qwen-3-8b-it (36 layers, 8B), Qwen-3-14b-it (40 layers, 14B), and Gemma-4-31b-it (60 layers, 31B, loaded in 8-bit quantization). They compared against surface baselines (concept-word dictionary, 25-topic model, TF-IDF with logistic regression), the same model's own answer to the question, and the fine-tuned classifiers published with the dataset.

The RFM algorithm computes the Average Gradient Outer Product (AGOP) matrix, which weighs which features are most important for telling the classes apart, and the top N = 5 eigenvectors from its eigendecomposition become the concept vectors. Each sentence was wrapped in a fixed statement guiding the model toward the concept. Scores were computed as cosine similarity between activations and concept vectors, averaged across layers, then across concept vectors, then min-max scaled to [−1, 1]. A threshold was swept on the reported split so the RFM accuracy and F1 in Tables 3 to 5 are the best attainable on that split rather than the value a frozen threshold would produce. Data were split into 1,000 training / 500 validation / 500 test sentences.

Linear probing used a stacked generalisation architecture: a separate RidgeClassifier with strong L2 regularisation trained on each layer's activations (every hidden layer except the earliest), producing per-layer decision scores; then a logistic regression meta-learner trained on aggregated per-layer scores. A nested 5-fold stratified cross-validation procedure was used to avoid data leakage. Three token pooling strategies were compared: last-token activation, mean pooling, and max pooling.

The human-annotated ESG dataset of Schimanski et al. (2024) contains 2,000 sentences per pillar (Environmental, Social, Governance), drawn from annual reports, responsibility reports, sustainability reports, and news articles; each sentence has a binary label (0 = domain not present, 1 = present). Inter-annotator agreement ranged from Fleiss' kappa 0.867 (Governance) to 0.926 (Social). Label distributions: Environmental 658 positive / 1,342 negative; Social 804 / 1,196; Governance 538 / 1,462.

Across all pillars, linear probing achieves the highest results of the activation-based methods and comes close to the fine-tuned models from Schimanski et al. (2024).

  • Environmental: best probe (last-token pooling on Qwen-3-8b) scored 0.951 accuracy, coming within 0.6 percentage points of the fine-tuned EnvRoBERTa, which achieved 0.957.

  • Social: best probe (max pooling, Qwen-3-14b) scored 0.924, statistically tied with the embedding baseline (0.925); the fine-tuned SocRoBERTa scored 0.934—a gap of 1.0 point.

  • Governance: best probe (max pooling, Qwen-3-8b) scored 0.876, versus 0.897 for GovDistilRoBERTa—a gap of 2.1 points.

The probe outscores the same model's own answer to the question in eleven of twelve comparisons, by 0.043 AUC on average, with the single exception being a tie (0.975 vs 0.978 for Llama-3.1-8B on Environmental). The authors conclude: reading the activations recovers concept content the output does not express.

Contrary to the authors' expectation that RFM concept vectors would transfer best from Beaglehole, Radhakrishnan, et al. (2026), the simpler linear probe was consistently the best activation-based method. RFM best configurations scored 0.946 (Environmental), 0.886 (Social), and 0.866 (Governance), all with Gemma-4-31b-it and the first concept vector. The comparison held even when both methods were given identical data (same 1,000 training and 500 test sentences): Linear probing remains ahead in eleven of the twelve model-pillar combinations, by 2.9 accuracy points on average, so the ordering reflects the extraction method rather than the evaluation protocol. For RFM, the first concept vector beats the mean of the top five concept vectors on accuracy in eleven of the twelve model-pillar combinations, a counterintuitive result the authors suggest may be because the additional vectors capture more peripheral concepts, which adds noise. Larger models performed better for RFM, with Gemma matching or outperforming all others on accuracy across every pillar.

Dictionary counts and topic proportions... sit far below both: a concept-word dictionary reaches 0.128 F1 on Governance against 0.757 for the best probe, and a 25-topic model reaches 0.373. TF-IDF with logistic regression was a harder baseline (0.894 AUC on Governance, above every RFM configuration there), though even there they trail the best probe by 2.0 accuracy points... and on Environmental and Social the gap widens to 6.9 and 6.0 accuracy points. Thus the surface-versus-judgment distinction... holds against the measures that count concept words, and not against a classifier free to learn whatever lexical cues the labels happen to carry.

  • Pooling strategy inconsistency: There is no single best pooling strategy—last-token was best on Environmental; max pooling on Social and Governance. The authors suggest the easier Environmental task needs only the last token's summary, while harder tasks benefit from max pooling's emphasis on strong signals.

  • Stacked architecture value: the stack earns more, up to five points on the harder Governance pillar, but gaps below one AUC point on Environmental—an easy concept needs one well-chosen intermediate layer, and stacking is worth its cost only when the concept is hard.

  • Wrapper sensitivity: "The wrapper governs the quality of the extracted concept vectors. Holding model, dataset and method fixed, changing the wrapper moves accuracy across a 9.4-point range, and the best wrapper shifts from one model to the next. Ending the wrapper with Final answer: rather than Answer:" moved accuracy by 2.6 points.

  • Continuous RFM score: Beyond classification, RFM yields a continuous score intended to reflect how strongly a concept is present in a text, whose validation awaits graded labels. Score distributions separated classes most clearly on Environmental and least on Governance, matching accuracy. Some confidently misclassified sentences fall on sentences whose label is itself arguable—e.g., We are committed to creating a successful, sustainable business that is economically, socially and environmentally conscious, which the dataset labels negative despite the method scoring it high on Environmental. Settling such cases needs re-annotation by experts.

  • Limitations: Gemma probing could not run due to GPU out-of-memory on the available hardware; on a card with enough memory... the same pipeline completes in minutes and places Gemma at or just below the best Qwen configuration on every pillar. Wrapper search covered only six wrappers, so how much better either method could do under an optimal wrapper is unknown, which weighs more heavily on RFM than on the probe. ESG is a niche subset within the financial sector, and Environmental sentences consistently score higher across all methods than Governance sentences, likely because environmental topics appear more often in training data. Current ESG datasets have only binary labels, so validating that graded reading is left to future work.

This study asked whether the internal activations of a frozen LLM can measure concept content in text without task-specific fine-tuning. They can, and cheaply. The paper's three contributions are: (1) a linear probe on frozen activations measures ESG content within 0.6 to 2.1 accuracy points of domain-specific fine-tuned models, beats the model's own answer in 11 of 12 model-pillar combinations, and needs a few minutes of GPU time per pillar (under four minutes for activations and RFM fitting on a single H100, about five for the full nested probe); (2) the concept vectors yield a continuous score that classification alone does not, whose validity awaits ordinal labels; and (3) the wrapper is identified as the least controlled step in the pipeline, moving results by several points. "For practitioners, the lightweight probe is the default choice: it is the strongest method we tested and among the cheapest to run. The concept-vector method earns its extra machinery only when the continuous score is itself the object of interest. Future work should develop a more principled way to design the wrapper" and validate the continuous score against graded labels.

Improvements for AI systems

Improvement 1: Cheap, fine-tune-free concept measurement via stacked linear probes.

Replace task-specific fine-tuning with a nested linear probe (RidgeClassifier per layer + logistic regression on layer scores) over frozen LLM activations.

What the improved system can do: Detect whether a text is about a given concept (e.g., ESG pillars) to within 0.6–2.1 accuracy points of domain-fine-tuned models, in 5 minutes of GPU time per concept, without retraining or altering the LLM. It can be rolled out to new domains or new concepts instantly from a few hundred labeled sentences.

Improvement 2: Activation-based judgment that overrides the model's stated output.

Because probes beat the LLM's own answer in 11/12 comparisons (by 0.043 AUC on average), use internal activations rather than generated text for classification tasks where the model may be evasive, sycophantic, or refusal-prone.

What the improved system can do: Reliably detect concept presence even when the model's verbal answer is wrong, unhelpful, or hedged — making it useful for audit tools, content moderation, and compliance screening where you cannot trust the model's self-report.

Improvement 3: Automatic wrapper optimization for concept-vector extraction.

The wrapper (prompt template) moves accuracy across a 9.4-point range and the best wrapper changes per model; this is the least controlled step in the pipeline. Add a wrapper-search step that evaluates a small set of wrappers on validation data and selects the best one for the specific model and concept.

What the improved system can do: Deliver stable, near-optimal concept-vector quality without manual prompt engineering, and report how sensitive the score is to wrapper choice — flagging cases where the measurement is unreliable.

Improvement 4: Difficulty-adaptive layer selection in the probe stack.

The stacked architecture buys up to 5 accuracy points on hard concepts (Governance) but <1 AUC point on easy ones (Environmental). Use validation performance on the target concept to decide how many layers to probe and whether to stack at all.

What the improved system can do: Automatically allocate compute: on easy concepts it uses one well-chosen intermediate layer; on hard concepts it engages the full stacked pipeline — reducing GPU cost while preserving accuracy.

Improvement 5: Per-concept pooling-strategy selection.

No single token pooling strategy wins (last-token best on Environmental; max pooling best on Social and Governance). Select pooling per concept via validation.

What the improved system can do: Squeeze out additional accuracy on each task by treating pooling as a tunable hyperparameter instead of a fixed design choice, with gains of several points on harder concepts.

Improvement 6: Continuous concept-intensity scoring with ambiguity flagging.

RFM concept vectors provide a continuous score reflecting how strongly a concept is present, and the confidently-misclassified sentences are often ones whose labels are genuinely arguable. Build a system that emits both the classification and the continuous intensity, and routes high-confidence disagreements to human review.

What the improved system can do: Produce graded concept measures (not just binary labels) for ESG ratings, harm severity, or sentiment intensity; automatically identify sentences whose labels are questionable and queue them for expert re-annotation — improving dataset quality and exposing annotation errors.

Improvement 7: Model-and-method selection guidance.

The paper shows RFM benefits from larger models (Gemma-4-31b wins all pillars) while linear probes peak on mid-size models (Qwen-3-8b/14b). Build a selector that, given a budget and a concept, picks between probe and RFM and between model sizes.

What the improved system can do: Tell a practitioner: for a binary label, use a probe on Qwen-3-8b; for a continuous score, use the first RFM concept vector on Gemma-31b — saving experimentation time and making the compute/accuracy trade-off explicit.

Improvement 8: Robustness to lexical surface cues.

Probes beat TF-IDF logistic regression by 2.0–6.9 accuracy points and vastly beat dictionary/topic baselines (0.128–0.373 F1 vs 0.757+), meaning they capture judgment, not just word counts. Use this to build classifiers that are harder to game by surface keyword stuffing.

What the improved system can do: Distinguish text that is genuinely about a concept from text that merely contains concept-related words (e.g., a report that mentions sustainability once without actually addressing it) — resisting adversarial or superficial wording in ESG reporting, propaganda, or spam.

Sources

Related papers