Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions".

Jane: The gist The results distinguish convention decodability, sensitivity to convention information, and cooperative performance by comparing linear probes and action translation in an unfamiliar partner environment.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: The title itself suggests they are moving beyond just "knowing" what the partner is saying to actually seeing if that knowledge translates into doing something useful in a cooperative setting. They set up a controlled environment using Hanabi-derived settings where agents have different communication conventions, and the same hint can mean different things depending on who sent it.

Jane: So, they’re testing two key aspects: whether the model can decode what the partner is communicating, and whether that decoding actually results in cooperative decisions. It sounds like a very nuanced test of agency.

Lu: They use linear probes to see how well different parts of the model recover those conventions—specifically distinguishing between intent versus target information when looking at hidden representations. This gives them a way to measure the "knowing" part, as they call it in their setup.

Meng: But the paper immediately points out a problem: even when they can decode that convention information, receiving decisions don't consistently follow the sender’s convention, which means decoding doesn't automatically mean acting correctly. That’s a critical failure point for real-world application.

Lalam: I agree with Meng; it shows that the gap between understanding a signal and applying it to an action is still quite wide, even when the model has read the hidden states of the interaction history.

The paper's summary: Tom: The summary boils down to comparing linear probes against action translation interventions. They found that probing for partner convention information is much easier on the intent axis than on the target axis when looking at hidden representations, but receiving decisions don’t consistently follow the sender’s convention even if that information is available.

Jane: So, they’re saying the model gets a better read on what someone *wants* to say than what they *want* to do next, and that distinction is key. That makes sense because intent and action are different things in communication.

Lu: They then compare two ways of presenting this information: presenting it as a rule, or translating it directly into an action recommendation. The results show that rule statements only yield modest changes in cooperation compared to what happens when they translate the convention into a direct action suggestion.

Meng: That’s useful for deployment because if we can give the model a translation rather than just telling it "follow this rule," the performance jump is bigger on average, which suggests translating information is more effective than just reading it as a static rule.

Lalam: It reinforces what we've seen: action recommendations seem to have much stronger responses when compared to stated rules, which makes sense because we’re focusing on the actual output decisions.

The paper's improvements: Tom: The paper suggests a few ways to improve this. First, they look at activation transfers from oracle actions and non-oracle hint restatements as donors; they found that these donors actually improved intent accuracy on both action classes. That’s a clear mechanism for boosting the model’s ability to get the right intention.

Jane: An oracle action donor—basically feeding the model what the correct decision was—seems to give it a significant boost in understanding its goal, which is helpful because we know intent accuracy is important for cooperation.

Lu: They also found that combining "the hint type and the predicted convention" provides a much more reliable decoding than just looking at the resulting intent decision alone, reaching an accuracy of zero point eight one when they combine those two readouts outside the model.

Meng: Combining those two readouts sounds like a good way to refine the internal representation because it’s better than relying on just one piece of information, which is something we need when training systems to be robust.

Lalam: That combination suggests that while decoding the convention is possible, applying it effectively requires more than just reading the hidden state; external processing seems necessary for a reliable output.

Conclusion: Tom: So, to wrap up, they’re distinguishing between how well the model reads conventions, how sensitive it is to that information, and its actual cooperative performance by comparing linear probes against action translations in an unfamiliar partner environment. They found that convention readout, behavioral adaptation, and game score need separate evaluation to fully understand this.

Jane: It seems the main thing they’re saying is that simply having the internal representation of what the partner is doing isn't enough; you also have to figure out how much it actually pushes your final decision in line with their conventions.

Lu: The implication for research is that we can isolate these components and test them separately, rather than just looking at a single score like a game win rate.

Meng: From an engineering standpoint, this means we can target specific weaknesses—like the translation step or the intent accuracy—instead of trying to fix everything at once with one big fine-tuning run.

Lalam: I think it suggests that for future work, we need to focus on interventions that actively force the decision axis to align with what the partner is communicating rather than just hoping the underlying representation is enough.