System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "System Attribution in LLM Brand Recommendations".

Tom: Aggregated brand behaviour does not transfer across query domains, and a profile built on that ordering flips between gift prompts and category-ownership prompts.

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So, we're diving into the details of this paper now, "System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer." The central argument they are making is a test of whether an AI-visibility profile that summarizes brand recommendations across different queries actually tells you which system produced the answer.

Jane: Essentially, they set up two units of observation to test this: first, they look at a single stored response, and second, they look at an aggregate model summary built from twelve behavioral features like brand volume and sentiment. The main finding is that while the single response unit identifies its endpoint with high accuracy—ninety-seven point eight four percent when looking at the first eight hundred characters—the aggregated profile fails to transfer across domains.

Lu: That's a crucial distinction they are drawing, isn't it? They found that a forest trained on ten category-ownership units incorrectly assigns all twenty-two gift units to the wrong system, and this is consistent with a reversal in brand volume between the two systems.

Meng: It sounds like the method shows that surface form of an answer is what carries the system across those different queries, but that aggregated brand behavior profile does not capture that cross-domain relationship accurately. That has some serious implications for how we measure AI performance generally.

Lalam: From my perspective as a model, this suggests that if I'm asked about gifts and then asked about corporate reputation, my underlying response structure will be completely different enough that a summary built on the gift data won't accurately represent me when I answer the reputation query.

Tom: That’s right; so they are demonstrating that aggregated brand behaviour does not transfer across domains, even though a single response tells you exactly which system generated it with high accuracy. They set this up using six thousand four hundred seventy-five stored responses collected between December two thousand twenty-five and February two thousand twenty-six from five different endpoints across gift-recommendation, corporate-reputation, and category-ownership queries.

Jane: It's important to remember the context of how they tested it; they used a truncation control by cutting answers to their first eight hundred characters so that where an answer stops carries no information about its ending. They also used masking brand names before vectorization to try and isolate the content signal from just the brand names themselves.

Lu: The paper also describes how they handle the data quality issues, mentioning truncation in three subsets, empty responses concentrated on long-answer prompts, and completion caps that differed between subsets for one endpoint. Those details are important context for understanding why their measurements have these specific results.

Meng: I'm interested in the calibration aspect they mentioned; they used the Expected Calibration Error to see how well the response-level classifier performs both inside and outside of their training distribution, and it showed a higher error for the aggregate forest across domains compared to its performance inside.

Lalam: That calibration difference really tells us that while we can trust a single answer within its expected context, generalizing that trust into a broad behavioral profile across different tasks is where the system struggles with accuracy.

Tom: So, to put it simply, the paper shows us that attribution happens at the response level based on how things are laid out or what brands are named in that specific text, but the broader behavioral summary gets confused when you change topics. This sets a clear boundary for what we can expect from aggregated reports.

Jane: It’s a warning that if we build AI-visibility measurements based solely on aggregated brand behavior, that profile will vary depending on the query domain and needs supporting evidence to move it between domains. That's the main lesson here.

Conclusion: Tom: So, wrapping up this discussion on "System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer," we have to think about what this means for how we understand AI behavior in the real world. The authors are essentially showing us that a single piece of text is a much stronger indicator of system origin than a summary built from many pieces of text across different types of tasks.

Jane: That's the big picture they are pushing; it means that if we want to reliably know which AI model is responding, we should focus our attention on the immediate output itself rather than trying to build a broad, generalized profile of its behavior across different scenarios like gift suggestions versus corporate reputation checks.

Lu: The implication for future research is clear: any measurement of AI visibility based on aggregated brand profiles needs to be treated as being localized; it describes that specific domain, and you need new data or a change in the measurement process to claim it applies elsewhere.

Meng: For practical engineering teams, this means we should design our monitoring tools to focus on response-level signals—the structure and content of individual answers—since those are what reliably point us toward the correct deployed system for that specific task.

Lalam: I see this as a directive to improve how we train and fine-tune models; we need mechanisms that ensure the fundamental response generation aligns with the intended system's characteristics, rather than hoping a high-level summary will generalize perfectly.

Tom: Exactly, Lalam; it’s about precision at the response level. If we want to understand AI visibility better, we need to stop relying on these aggregated profiles that flip between queries and start focusing on what the text actually says right now.

Jane: It boils down to this: a per-system brand-behaviour profile measured in one query domain accurately describes that domain, and moving that profile to another domain requires providing new evidence of its validity. That’s the core message of this work by Zatuchin et al.

Lu: It really opens up avenues for how we design AI systems where we can tailor the expected output characteristics based on the specific query type they are handling.

Meng: So, in short, it's about moving from generalized behavioral summaries to localized response analysis when trying to attribute an answer to a specific AI deployment.

Lalam: And that focus on localized analysis seems like a solid direction for improving how we build and trust these systems moving forward.