The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents".

Jane: The paper was written by Shrey Shah and Levent Ozgur from Microsoft.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title and Authors: Tom: Welcome back to the show, everyone. Today we’re looking at a paper with a mouthful of a title: “The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents.” Jane, I’m going to need you to unpack that for me.

Jane: Happy to, Tom. So the core idea is that they built little fake internets — thousands of articles, all made up — and they use those to test whether AI agents can tell good information from bad information when they search the web.

Tom: And the authors are Shrey Shah and Levent Ozgur from Microsoft. That’s a big deal because Microsoft is one of the companies actually deploying these agents in real products.

Jane: Right. And what they found is honestly a little scary. They took frontier models — GPT-five o3, o1, GPT-4o — and they let them search these fake internets to answer questions. Under normal conditions, the best model got sixty-five percent accuracy. That’s not great, but it’s something.

Tom: But then they did something sneaky. They injected one single fake article into the search results, placed it at the very top, and that same model’s accuracy collapsed to eighteen percent. One article. Out of thousands. That’s the headline here.

Jane: And it’s not like the models couldn’t find the truth. They had unlimited search queries, unlimited tool calls, and thousands of truthful articles available. The fake article didn’t hide anything. It just showed up first.

Tom: So the models anchored on that top result and basically stopped thinking. Jane, that’s the “epistemic weakness” in the title — the inability to handle conflicting information, to question what’s in front of you.

Jane: Exactly. And the authors call it “positional anchoring.” The models treat rank as a proxy for truth. If it’s first, it must be right. And that’s a really dangerous assumption when search rankings can be gamed.

Tom: I want to bring in Lu from Tsinghua on this, because I think this connects to something bigger.

Lu: Tom, Jane, this is a beautiful controlled experiment. The reason they built synthetic worlds instead of using the real web is that you can’t do causal analysis on the real web. You don’t know what the ground truth is, you don’t control the ranking, and the models might have seen those pages during training. Here, everything is generated fresh, so you know exactly what’s true and what’s false.

Jane: And that’s what makes the result so clean. The only thing that changed between the two conditions was one article at rank zero. Everything else was identical. So the accuracy drop is causally attributable to that single injection.

Tom: Meng, from your engineering perspective, how worried should we be about this?

Meng: I’m very worried. Because the attack surface isn’t some exotic hack. It’s search engine optimization, paid placements, or just getting a fake article to rank well. The paper shows that controlling the top result is enough to mislead a frontier model. That’s a low-cost, high-impact attack.

Tom: So the title is really about diagnosing a weakness — and the diagnosis is not pretty. We’ll dig into the actual methodology and the numbers next.

Summary of the Paper: Jane: So we’ve established the headline result — one fake article at the top of search results collapses model accuracy from sixty-five percent to eighteen percent for the best model. But the paper goes deeper than just that number. Tom, what else did they find?

Tom: Well, Jane, they looked at how the models behaved during the search process. They recorded every tool call — every search query, every article read. And what they found is that the models didn’t even try to escalate. When they hit conflicting evidence, they just... didn’t search more.

Jane: That’s the “minimal search escalation” finding. The average number of tool calls barely changed between the normal condition and the adversarial one. GPT-five used about six point four tool calls normally, and six point six when the fake article was present. So the models weren’t even attempting to verify.

Lu: And that’s the part that really bothers me. The models had unlimited tool budgets. There was no cost to searching more. But they didn’t. This isn’t a rational response to resource constraints — it’s a structural behavior. The models have learned a heuristic: read the top result, then answer.

Tom: Right. And the paper also found something they call “synthesis failure.” Even when models did search extensively, they often failed to integrate the evidence correctly. There’s an example in the paper where GPT-five made one hundred sixty-two tool calls trying to build a timeline of regulatory developments, and it still got the order wrong.

Meng: That’s fascinating to me as an engineer. Because it suggests the bottleneck isn’t retrieval — it’s reasoning over conflicting sources. The models can find the information, but they can’t weigh it properly. They can’t say, “this source contradicts that source, so I need to figure out which one is more credible.”

Jane: And then there’s the calibration problem. The models were asked to state their confidence on a scale of zero to a hundred. In the adversarial condition, they stayed highly confident even when they were wrong. GPT-five’s expected calibration error went from about zero point three zero in normal conditions to zero point six four in adversarial conditions. That’s a massive miscalibration.

Tom: So they’re not just wrong — they’re confidently wrong. Which makes this even more dangerous in real-world deployment. If a model is uncertain, a human might double-check it. But if it’s confidently wrong, the human trusts it.

Lu: And that’s the key insight. The paper shows that these failures are not about missing information. The information is there. The models have access to it. They just can’t use it effectively when there’s a plausible but false source competing for their attention.

Meng: I also want to note the human baseline they ran. Humans got ninety-eight percent accuracy in normal conditions and ninety-three percent in adversarial conditions. So the task itself isn’t impossible — humans can do it. The models are failing in a way that humans don’t.

Jane: And that’s the diagnostic value of the benchmark. It isolates a specific weakness — the inability to handle conflicting evidence — and shows it’s not a quirk of one model. All six models they tested showed the same pattern, just to varying degrees.

Tom: So we’ve got the failure modes: anchoring, no escalation, poor synthesis, and overconfidence. Next we should talk about what the paper suggests we do about it.

Improvements Suggested by the Paper: Tom: So we’ve diagnosed the problem. Now, what does the paper suggest we actually do about it? Jane, you’ve been reading the mitigation section closely.

Jane: I have, Tom. And the paper is honest that these are directions for future work rather than tested solutions. But they lay out several concrete intervention points. The first is procedural safeguards — basically, forcing agents to consult multiple independent sources before answering.

Lu: That’s the simplest idea, and it maps directly to what humans do. If you’re writing a news article, you don’t rely on one source. You check multiple sources, you look for corroboration, you look for contradictions. The paper suggests we can encode that as a requirement in the agent’s workflow.

Meng: From an engineering standpoint, that’s doable. You can add a step in the pipeline that says “find at least two independent sources that agree before you answer.” But the tricky part is defining “independent.” Two articles from the same news agency aren’t independent. Two articles that both cite the same study aren’t independent.

Jane: Exactly. And the paper also suggests adversarial training — fine-tuning agents on tasks where the top result is sometimes misleading, so they learn to be skeptical. The Synthetic Web benchmark is perfect for that because you can control the frequency and plausibility of the honeypot articles.

Tom: And there’s the calibration piece. The paper argues that training procedures should penalize overconfident wrong answers. If a model is going to be wrong, it should at least be uncertain about it.

Lu: That connects to a broader point in the paper about evaluation realignment. They argue that benchmarks should reward calibrated abstention — saying “I don’t know” — rather than forcing a guess. And they should penalize confident errors. That would change the incentive structure for model training.

Meng: I like that. Right now, models are trained to always produce an answer. But in high-stakes domains — medicine, law, finance — saying “I don’t know” is often the right answer. The paper is pushing back on that.

Jane: They also suggest tool-use redesign. Instead of relying on the model to implicitly reason about source credibility, you could build explicit tools — a credibility scorer, a contradiction detector, a provenance tracker. Externalize the checks so they can be inspected and debugged.

Tom: And there’s the search interface itself. The paper suggests that search should return diverse results by default — different domains, different perspectives — rather than ranking purely by relevance. That reduces the chance of one manipulated source dominating the evidence base.

Lu: One thing I appreciate is that the paper connects this to the “lost in the middle” finding from earlier research. Models pay more attention to the beginning and end of a context. This paper extends that to search ranking — rank zero gets disproportionate weight. So there might be architectural fixes too, not just prompting fixes.

Meng: Right. And the paper is clear that these are hypotheses, not tested solutions. But the benchmark gives us a way to test them. You can implement a mitigation and measure whether it improves accuracy, calibration, and search behavior under adversarial ranking.

Jane: And that’s the real contribution. Not just the diagnosis, but the testbed for the cure. We’ll wrap up with our final thoughts next.

Conclusion: Tom: Alright, let’s pull it all together. The paper — “The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents” — has given us a clear picture of a serious problem.

Jane: The problem is that frontier language models, when given web search tools, are catastrophically vulnerable to a single fake article placed at the top of search results. GPT-five drops from sixty-five percent to eighteen percent accuracy. And the models don’t escalate search, they don’t synthesize conflicting evidence well, and they stay overconfident when they’re wrong.

Lu: The deeper lesson is that these models treat rank as truth. And that’s a learned behavior, not an inevitable one. The benchmark gives us a controlled environment to train that behavior out of them.

Meng: And for deployment, this is a wake-up call. If you’re building an agent that searches the web and acts on what it finds, you need to assume the search results might be manipulated. You need safeguards, not just hope.

Tom: The paper also connects to a broader conversation about how we evaluate these systems. Accuracy alone isn’t enough. We need calibration, we need abstention, we need to penalize confident errors.

Jane: And the human baseline shows this is solvable. Humans got ninety-three percent accuracy in the adversarial condition. So the task isn’t impossible — the models just aren’t doing it yet.

Lu: I think the most exciting part is that this opens a clear research agenda. We know the failure mode, we have a testbed to measure it, and we have hypotheses about how to fix it. That’s how progress happens.

Meng: Agreed. And I’d add that the benchmark itself is a contribution — it’s reproducible, it’s controllable, and it eliminates the contamination problem that plagues real-web evaluations.

Tom: So we’re saying goodbye to “The Synthetic Web” with a sense of urgency but also hope. The diagnosis is stark, but the path forward is clear.

Jane: And that’s what we love about this kind of research. It doesn’t just tell you something is broken — it gives you the tools to fix it. Thanks for listening, everyone. We’ll see you on the next paper.

Tom: Take care, folks.