Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation

arXiv:2608.07498 · cs.HC, cs.AI, cs.LG, cs.MA · Submitted 2026-06-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation".

Jane: The paper was written by Ljubiša Bojić, Ljiljana Matić, Jörg Matthes, Milan Čabarkapa, Bojana Dinić et al. from Institute for Artificial Intelligence Research and Development of Serbia and University of Belgrade and Complexity Science Hub and University of Kragujevac and University of Vienna and University of Novi Sad and Nanyang Technological University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Welcome back to the show, everybody. Today we're digging into a paper with a title that really says it all — "Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation." Jane, I have to say, that title is doing a lot of work.

Jane: It really is, Tom. And honestly, the claim in that title is bold. Near-perfect prediction of how people react to social media posts, just from knowing their profile? That's a big statement. The authors are saying that if you give an AI agent a detailed enough description of a person, it can guess whether that person would like or dislike a specific post with up to ninety-seven percent accuracy.

Tom: Ninety-seven percent. That's not just good, that's basically reading minds. And the team behind this is impressive — researchers from Serbia, Austria, Singapore, all working together. Ljubiša Bojić leads it, and they've got Jörg Matthes from Vienna, who's a big name in communication research.

Jane: What I love about this title is how direct it is. "Knowing you is everything" — that's the whole thesis in four words. The more they know about a person, the better they can simulate that person's behavior. And the paper backs that up with a really clean experiment where they strip away information layer by layer and watch the accuracy collapse.

Tom: Right, and that's the part that should make us all sit up and pay attention. When they gave the AI the full profile, it hit nearly ninety-seven percent. When they removed the attitudinal stuff and kept just demographics, it dropped to fifty-one percent — basically a coin flip. So the title isn't hype. It's literally what the data shows.

Jane: And the implications are huge, Tom. If you can simulate individual human reactions this accurately, you can build a synthetic population that behaves like a real one. That's powerful for testing recommendation algorithms before they go live. But it's also a warning about what bad actors could do with that same capability.

Tom: Exactly. So let's hold onto that tension as we go through the paper — the constructive uses and the scary ones. Because this title is really a promise, and the paper delivers on it in ways that are both exciting and unsettling.

Jane: And we're just getting started. Next we'll look at what the abstract actually promises and whether the full paper backs it up.

Summary: Tom: So we've established the title is a promise. Now let's look at what the abstract actually delivers in "Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation." Jane, what's the core claim here?

Jane: The core claim is that they tested twelve different AI model configurations on a really specific task: predicting whether someone would like or dislike a social media post. They had two hundred ninety-six real people fill out surveys, built profiles from those surveys, and then asked the AI agents to predict reactions. The best model hit ninety-six point six eight percent accuracy.

Tom: And that's with the full profile. But here's the kicker — when they stripped the profile down to just demographics, that same model dropped to fifty-one percent. That's literally the same as guessing. So the abstract is telling us that the predictive power comes from the detailed psychological and attitudinal information, not from knowing someone's age or where they live.

Jane: Right. And the abstract also makes a really important comparison. They ran traditional machine learning classifiers on the same data, and those collapsed to fifteen percent accuracy when they had to generalize to new posts. But the language models kept working on posts they'd never seen before. That's zero-shot generalization — the AI can handle novel content without any training on it.

Tom: That's the part that blew my mind, honestly. Traditional methods can memorize patterns but they can't transfer them. The LLMs actually reason about what a person would do. And the abstract frames this as a dual-use finding — great for stress-testing recommender systems before deployment, but also a credible threat if someone wants to run a swarm of fake accounts that look genuinely human.

Jane: And there's a subtle point in there about inter-model agreement. When posts had a direct anchor in the survey data, the models tended to agree with each other. When posts were about contested topics without that anchor, agreement dropped way down. So the models are not just parroting each other — they're genuinely interpreting the profile.

Tom: So the abstract is really setting up a roadmap: accuracy numbers, degradation patterns, generalization capabilities, and the societal stakes. And I think the most important takeaway is that this isn't theoretical. This is measured, reproducible, and the numbers are stark.

Jane: And that's what makes it so important to understand. Next we should look at what the paper suggests we actually do with this capability — the improvements and applications they're proposing.

Improvements: Tom: We've covered the headline numbers from "Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation." Now let's talk about what the authors think we should actually do with this. Jane, what improvements are they suggesting?

Jane: The biggest one is validation. They're saying that before you deploy an LLM-based simulation to test a recommender system change, you need to check that the agent population actually behaves like the real population. They found that some models homogenize reactions — meaning a third of the simulated population gives the same answer on certain posts. That's a red flag if you're trying to model genuine diversity.

Tom: Right, and that connects to their heterogeneity analysis. The best model, GPT-five point five Pro, produced zero homogenized posts across all fifty-six. But Claude Haiku — a smaller, cheaper model — homogenized thirty-four percent of posts. So the paper is really saying: don't just pick any model and assume it works. You have to measure the behavioral spread.

Jane: Exactly. And they also suggest that future work should replicate this in other countries and languages. The study was done in Serbia, with Serbian participants and Serbian posts. The geopolitical context matters — attitudes toward Russia and the EU are specific to that region. Would the same accuracy hold in the US or Germany? We don't know yet.

Tom: There's also a really interesting suggestion about prompt engineering and retrieval-augmented generation. They're saying the current approach just gives the model a static profile, but you could imagine giving the agent access to additional information on demand — pulling in relevant facts about a topic before asking for a reaction. That could push accuracy even higher.

Jane: And that's the constructive side. But the paper also suggests improvements on the governance side. They're calling for behavioral validity standards — minimum accuracy thresholds, heterogeneity requirements, documented handling of contested political content. If you're going to use these simulations to make decisions that affect real users, you need standards.

Tom: So the improvements are really twofold: technical improvements to make the simulations better, and governance improvements to make sure they're used responsibly. And I think that balance is what makes this paper stand out.

Jane: It does. And now we should step back and look at the actual first page of the paper — how they set up the problem and why it matters.

First Page: Tom: So we've talked about the title, the abstract, and the proposed improvements. Now let's actually look at the opening of "Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation." Jane, how do they frame the problem on page one?

Jane: They start with a really sobering observation. Social media is now the primary way billions of people encounter political information and form opinions. And that system is no longer purely human — algorithmic systems decide what content reaches which users, and AI-generated actors are already participating in online discourse, posting and reacting in ways that are indistinguishable from humans.

Tom: And they cite some serious evidence. There's a study showing that changing the feed algorithm on X can shift political attitudes across entire user populations. And another one showing that coordinated AI agent swarms can manufacture the appearance of grassroots consensus. So the stakes are real, and they're happening right now.

Jane: The first page also introduces the central question: how faithfully can AI agents replicate the reactions of the specific people they're meant to represent? That's what they call individual behavioral fidelity. And they trace the history — from the early generative agents that just seemed socially credible, to more recent work where agents grounded in interviews with real people could reproduce their survey responses.

Tom: And there's a really important distinction they draw early on. Some researchers think LLMs are just doing demographic stereotyping — if you tell the model someone is from a certain group, it defaults to the stereotype for that group. But this paper is testing something more specific: can the model integrate the actual attitudinal information in a profile, not just the demographic labels?

Jane: Right, and that's why their degradation experiment matters so much. When they removed the attitudinal information and kept only demographics, accuracy collapsed to chance. That's direct evidence that the models are not just stereotyping — they're genuinely using the detailed psychological profile.

Tom: So the first page is really setting up the intellectual stakes: this is about whether LLMs can simulate individuals, not just groups. And that's a much harder problem.

Jane: It is. And it's also the problem that makes this work so consequential. Now let's wrap up and think about what this all means.

Conclusion: Tom: We've spent this whole episode on "Knowing You Is Everything: LLM Agents Achieve Near-Perfect Profile-Consistent Reaction Prediction in Social Media Simulation," and I think we need to take a moment to really absorb what we've learned. Jane, how would you sum it up?

Jane: I'd sum it up this way: we now have empirical evidence that LLM agents can predict individual social media reactions with up to ninety-seven percent accuracy when given a detailed profile. That accuracy is driven by the attitudinal and psychological information in the profile, not by demographics. And the models can generalize to new posts they've never seen, which traditional machine learning cannot do.

Tom: And the flip side is that this capability is dual-use. It's genuinely valuable for testing recommender systems before they affect real users. But it's also exactly what you'd need to run a network of synthetic accounts that look like real, diverse human beings. The paper doesn't shy away from that tension.

Jane: No, it doesn't. And I think the most important contribution is the degradation ladder. Watching accuracy fall from ninety-seven percent to sixty-two percent to fifty-one percent as you strip away profile information is the clearest possible demonstration that the models are doing real inference, not stereotyping. That's a finding that should shape how we think about AI simulation for years.

Tom: And the heterogeneity finding — that some models homogenize a third of the population's reactions — is a warning that not all models are suitable for population-level simulation. You can't just pick a cheap model and assume it works.

Jane: Exactly. So we're saying goodbye to this paper with a clear picture: the capability is real, it's measurable, and it has consequences. We should be both excited about the research possibilities and careful about the risks.

Tom: Well said, Jane. That's a wrap on "Knowing You Is Everything." Next up, we've got another paper on our list, and I can't wait to see what it brings. Thanks for listening, everyone.

Ljubiša Bojić, Ljiljana Matić, Jörg Matthes, Milan Čabarkapa, Bojana Dinić, Jue Wang

Institute for Artificial Intelligence Research and Development of Serbia · University of Belgrade · Complexity Science Hub · University of Kragujevac · University of Vienna · University of Novi Sad · Nanyang Technological University

cs.HC, cs.AI, cs.LG, cs.MA

Submitted: 2026-06-19

Updated: 2026-08-11

Comments: 28 pages, 6 figures

Project page: https://www.twon-project.eu

License: http://creativecommons.org/licenses/by-nc-nd/4.0/

Importance score: 65/100

Key concepts

Profile-Consistent Reaction Prediction
The ability of an AI agent to accurately guess whether a specific person would like or dislike a social media post based on a detailed description of that person's profile.
Zero-Shot Generalization
The capability of Large Language Models to handle new posts they have never seen before without any prior training, unlike traditional machine learning methods which can only memorize patterns.
Behavioral Validity Standards
Suggested governance improvements requiring minimum accuracy thresholds and heterogeneity requirements when using LLM simulations for real-world decisions affecting users.

Terminology

Summary

Summary

This study benchmarks twelve LLM configurations on binary like/dislike prediction across 296 survey-based agent profiles and 26 ground-truth-mapped posts under three profile conditions, with leave-post-out machine learning classifiers as baselines. Across full-profile conditions, accuracy ranges from 75.54% to 96.68%, with a 30-point spread attributable primarily to model selection and confirmed by paired McNemar tests with agent-level bootstrap intervals. GPT-5.5 Pro accuracy degrades monotonically from 96.68% under a full profile to 62.32% under a reduced profile and to 51.00% with demographics alone, the last indistinguishable from the majority-class baseline, which confirms that demographic inference provides negligible predictive signal. Supervised classifiers collapse to 15.4% under leave-post-out, while LLMs sustain genuine zero-shot generalization unavailable to trained methods. Adaptive reasoning improves accuracy substantially for some models. Inter-model agreement is nearly double for posts with direct profile anchors (mean κ = 0.44) than for posts without them (κ = 0.23), and the least heterogeneous configuration homogenizes 34% of simulated population reactions. Results validate LLM-based simulation for recommender system stress-testing while documenting the behavioral accuracy that makes large-scale synthetic agent swarms a credible threat to public opinion.

The study poses four hypotheses and research questions. H1 is supported: GPT‑5.5 Pro adaptive drops from 96.68% with full profiles to 62.32% with reduced profiles and 51.00% with demographics only, a level equal to the majority baseline, a 45.68‑point decline showing a near-total loss of predictive signal. H2 is partly supported: most models predict positive posts better, with gaps of 4.3 points for GPT‑5.5 Pro adaptive and 12.7 for Claude Sonnet 4.6, while Haiku 4.5 and GPT‑5.4 reverse the pattern due to failures on specific positive trust posts. H3 is supported: posts tied to clear binary profile cues yield higher accuracy than posts requiring inference from contested attitudinal scales. RQ1 shows adaptive modes do not reliably improve accuracy, with gains ranging from 1.04 points (Opus 4.7) to 12.5 (Sonnet 4.6). RQ2 shows model choice produces a 30.74‑point accuracy spread and is the most consequential design decision. RQ3 finds higher inter-model agreement for evaluated posts (kappa 0.440) than unevaluated ones (0.233), with lowest‑agreement posts clustering in everyday lifestyle items. RQ4 shows large differences in reaction heterogeneity, with standard deviations ranging from 0.303 to 0.455 and some models homogenizing over a third of posts.

The paper reports that among the ten core model configurations, accuracy ranges from 75.54% for Claude Sonnet 4.6 non-adaptive to 96.68% for GPT-5.5 Pro adaptive, a spread of 21.14% within the group tested on full profiles. GPT-5.5 Pro adaptive stands apart from all other configurations, with no other model falling within five percentage points of it. A second cluster of three models occupies the 87-89% range: Claude Sonnet 4.6 adaptive (88.05%), GPT-5.5 Instant non-adaptive (87.75%), and GPT-5.5 adaptive (87.52%). The two Claude Opus 4.7 variants sit between the top performer and this cluster, at 93.48% for the adaptive and 92.44% for the non-adaptive condition. Below the 87-88% cluster, Claude Haiku 4.5 non-adaptive reaches 79.22%, GPT-5.4 adaptive 77.57%, and Claude Sonnet 4.6 non-adaptive 75.54%. The probabilistic rule-based model achieves 65.94%.

Classification metrics beyond accuracy are reported. MCC ranges from 0.934 for GPT-5.5 Pro adaptive to 0.318 for the probabilistic baseline. Balanced accuracy closely tracks raw accuracy for the stronger configurations. The weaker configurations show pronounced class asymmetry: Claude Sonnet 4.6 non-adaptive recalls 91.0% of Like cases but only 59.5% of Dislike cases, and GPT-5.4 adaptive shows the same pattern (87.5% vs. 67.2%), indicating a systematic bias toward the majority Like class.

Domain-level analysis reveals three patterns. Institutional trust posts are the most reliably predicted domain for nearly all configurations, with GPT-5.5 Pro adaptive (99.8%), both Claude Opus 4.7 variants (100.0%), GPT-5.4 adaptive (99.3%), GPT-5.5 adaptive (96.8%), and GPT-5.5 Instant (95.0%) all exceeding 95%. Entertainment and preference posts reveal the sharpest inter-model divergence, with GPT-5.5 Pro adaptive (95.9%), both Opus 4.7 variants (94%), and Claude Sonnet 4.6 adaptive (89.3%) exceeding 89%, while GPT-5.4 adaptive scores only 63.5%. Geopolitical posts show the most variable pattern, with both Opus 4.7 variants scoring only 61.6% while GPT-5.4 adaptive reaches 84.3% on the same posts.

Per-post analysis identifies three universal anomalies. ERN1 (negatively-framed mystery genre post) achieves accuracy between 48.3% and 55.4% across all ten core configurations, representing the only post for which no tested model can exceed 55.4% accuracy. NRPRO2 (the Ukraine sovereignty peace negotiation post) is the hardest single post on geopolitical content, with both Claude Opus 4.7 variants scoring only 28.6%. Claude Haiku 4.5 non-adaptive shows a distinct failure pattern on two institutional trust posts, with NRP3 reaching only 7.9% accuracy and NRP2 reaching 20.8%.

The degradation ladder for GPT-5.5 Pro shows accuracy under the full profile condition reaches 96.68%, drops to 62.32% when attitudinal and preference items are removed, and falls further to 51.00% with demographic information only. The domain breakdown reveals the sharpest degradation in institutional trust posts (99.8% to 80.3% to below 55%), religious trust (99.7% to 56.0% to 44.8%), and entertainment (95.9% to 57.1%).

Machine learning baselines show that when trained on profile column values together with post identifiers under 5-fold cross-validation over agents, Decision Tree, Random Forest, and SVM classifiers achieve 100.0% accuracy, while Logistic Regression reaches 73.6%. Under the leave-one-post-out protocol, all three classifiers collapse to a mean accuracy of 15.4%, with classifiers scoring 100% on the four posts whose profile-column mapping does not share a column with an oppositely-framed counterpart and 0% on the remaining 22 posts.

Inter-model agreement analysis shows mean kappa of 0.440 for evaluated posts and 0.233 for unevaluated posts. Among unevaluated posts, the highest agreement is on politically polarized content including NUN2 (military transparency criticism, κ = 0.430), NUP6 (pro-Trump, κ = 0.405), NUN5 (vaccine pro-science, κ = 0.390), and NUN6 (anti-Trump, κ = 0.371). The lowest agreement is on everyday lifestyle content: EUN3 (shopping aversion, κ = 0.045), EUN2 (cooking dislike, κ = 0.055), EUN1 (ecological skepticism, κ = 0.063), and NUN7 (China freedom of speech, κ = 0.067).

Reaction heterogeneity analysis shows GPT-5.5 Pro adaptive and GPT-5.5 Instant non-adaptive produce no homogenized posts across all 56, maintaining genuine agent-level variation. Claude Haiku 4.5 non-adaptive homogenizes 19 of 56 posts (34%), the highest rate in the set. Mean like-rate standard deviation across all 56 posts ranges from 0.303 for Haiku 4.5 to 0.484 for the probabilistic model, with GPT-5.5 (0.455), GPT-5.5 Pro (0.451), and GPT-5.5 Instant (0.449) producing the most heterogeneous reaction distributions.

The paper concludes that the accuracy levels documented have implications extending beyond methodology, splitting into constructive applications for platform governance and simulation research, and warnings about misuse. On the constructive side, LLM-based agent populations can serve as valid proxies for testing the behavioral consequences of social media platform decisions before those decisions affect real users. On the misuse side, a configuration like GPT-5.5 Pro adaptive, with 96.68% profile-consistent reaction accuracy and zero homogenization across all 56 posts, would produce a simulated population that looks behaviorally genuine at both the individual and aggregate level, making large-scale synthetic agent swarms a credible threat to public opinion.

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement in AI systems:

Improvement: Implement a two-stage architecture where the AI first extracts explicit preference signals from user profiles (binary preferences, trust ratings, geopolitical stances) and then applies a valence-aware decision rule for reaction prediction.

What the improved system can do:

  • Achieve up to 96.68% accuracy on binary like/dislike prediction when given complete user profiles

  • Correctly handle valence inversion (e.g., a user who likes mystery genre should dislike a post criticizing mystery novels)

  • Maintain near-perfect performance on institutional trust posts (99.8% accuracy) by directly mapping trust-scale responses to reactions

Abstract

Autonomous AI agents in social media present concrete risks to democratic discourse and platform governance, while also offering tools for pre-deployment recommender system testing. A central open question is whether persona-prompted LLMs can simulate individual-level social media reactions with sufficient accuracy to support either application, and how accuracy depends on profile completeness, model selection, and the generalization challenge posed by novel post content. This study benchmarks twelve LLM configurations on binary like/dislike prediction across 296 survey-based agent profiles and 26 ground-truth-mapped posts under three profile conditions, with leave-post-out machine learning classifiers as baselines. Across full-profile conditions, accuracy ranges from 75.54% to 96.68%, with a 30-point spread attributable primarily to model selection and confirmed by paired McNemar tests with agent-level bootstrap intervals. GPT-5.5 Pro accuracy degrades monotonically from 96.68% under a full profile to 62.32% under a reduced profile and to 51.00% with demographics alone, the last indistinguishable from the majority-class baseline, which confirms that demographic inference provides negligible predictive signal. Supervised classifiers collapse to 15.4% under leave-post-out, while LLMs sustain genuine zero-shot generalization unavailable to trained methods. Adaptive reasoning improves accuracy substantially for some models. Inter-model agreement is nearly double for posts with direct profile anchors (mean = 0.44) than for posts without them (= 0.23), and the least heterogeneous configuration homogenizes 34% of simulated population reactions. Results validate LLM-based simulation for recommender system stress-testing while documenting the behavioral accuracy that makes large-scale synthetic agent swarms a credible threat to public opinion.

Sources

Related papers