AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning".

Jane: Here's the gist of the paper. The gist The benchmark AGORA introduces an archive-grounded, cross-domain test for agentic workplace document reasoning, finding that even the strongest models only achieve 59.4% accuracy across eight domains because true reasoning requires deliberate exploration rather than exhaustive scanning<ref:2606.24526#pg12> How it works AGORA is designed to stress archive-groundedness, agentic exploration, and cross-domain coverage by pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens<ref:2606.24526#pg12> The benchmark requires agents to locate sparse evidence across a large, messy collection of workplace files, reconcile inconsistent terminology, units, and time conventions, and compute an answer<ref:2606.24526#pg4> This setting forces agents to explore deliberately rather than scan exhaustively because each collection is far larger than the context window of current models<ref:2606.24526#pg12> Benchmark Composition The AGORA benchmark is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering It consists of eight source collections, one per domain, each paired with a set of natural language queries answered over that collection The four formats collected—PDFs converted to Markdown via dots.ocr, Excel sheets as compact table profiles, CSV files parsed as single sheets, and plain-text Markdown files—are normalized into plain text so a language agent can read and reason over them directly Each query is cross-document and multi-hop, meaning its evidence is sparsely scattered across several files amid much unrelated material Evaluation and Results The evaluation involves running eight proprietary and open-weight models on AGORA inside a minimal harness called mini-swe-agent, which exposes only a bash tool The results show that the task is far from solved, as even the strongest model reaches only 59.4% overall accuracy Performance splits into two tiers, with a frontier tier of five models occupying a 40–60% band and the remaining three falling well below it Per-domain accuracy varies substantially, showing that difficulty is often not a property of the benchmark alone but emerges from the interaction between a specific model and a specific domain Construction Pipeline The AGORA construction pipeline involves three stages: 1.

Tom: Document collection and preprocessing, which uses deep search to retrieve semantically relevant web documents and produces a candidate list that is then verified and downloaded Documents are chunked into 8,000-token sliding windows for fine-grained retrieval by format-specific rules, and seed chunks are selected using an information-density heuristic 2.

Jane: Task synthesis, where an agent retrieves cross-document questions and progressively enhances them through refinement and obfuscation This involves drafting tasks grounded in seed chunks, followed by refinement to enforce consistency and naturalness, and obfuscation to remove residual leakage 3.

Tom: Quality control, which applies difficulty filtering and multi-check verification This stage includes discarding tasks solvable from parametric knowledge alone and assessing surviving tasks by a threemodel panel to guarantee sufficient difficulty Limitations The paper notes that AGORA spans eight professional domains distilled from official occupational classification systems, which is broader than prior single-source benchmarks but not exhaustive of all workplace settings The authors discard tasks solvable from parametric knowledge alone via a closed-book filtering stage to ensure current models must genuinely reason over the mounted collection Furthermore, all models are evaluated inside a single minimal harness exposing only a bash tool to isolate model capability from scaffolding engineering The paper also acknowledges that pretraining corpora may absorb more documents of this kind, potentially eroding the guarantee of the closed-book filtering safeguard against knowledge leakage over time The AGORA benchmark is a cross-domain test for archive-grounded agentic document reasoning, pairing 362 verifiable numeric queries with eight domain collections of 9,664 documents and 372M tokens Built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering with human verification, AGORA jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage Evaluating eight models on AGORA reveals that even the strongest reaches only 59.4% accuracy and exhibits systematic per-domain weaknesses The results show that difficulty is often not a property of the benchmark alone but emerges from the interaction between a specific model and a specific domain AGORA serves as a rigorous, reproducible testbed for the next generation of document-reasoning agents The benchmark is constructed to meet four desiderata: Archive-groundedness, Cross-domain coverage, Agentic exploration and evidence integration, and Verifiable evaluation The final task review process involves launching two subagents—one blind and one with a reasoning chain—to determine the reliability of the task before human annotation AGORA is intended to drive progress in agents that can autonomously explore and reason over large document archives The data used in this work is derived from publicly available sources, which do not contain private personally identifying information The benchmark is designed to test whether LLM agents can perform archive-grounded reasoning in realistic workplace settings AGORA is far from saturated, and performance splits into two tiers, with no model exceeding 60% accuracy The lower tier models often approach the floor domain by domain, and smaller models are effectively non-functional on AGORA The paper concludes that archive-grounded document reasoning remains far from solved for current models Page 1 --- AGORA is short for Archive-Grounded Office Reasoning Assessment Agri Arch Biz Edu Fin Health Law Tech 8% 13% 6% 8% 31% 11% 8% 13%> Page 2 --- Benchmark Corpus Test Multi-file Archive-grounded Agentic HotpotQA (Yang et al.,2018) Wikipedia MuSiQue (Trivedi et al.,2022) Wikipedia TAT-QA (Zhu et al.,2021) Financial reports FinanceBench (Islam et al.,2023) U.S.

Title and authors: Jane: So starting out, let's look at the title of this paper, AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning. It’s a mouthful because it really pinpoints what they are testing.

Tom: Yeah, it sounds like they are building a test to see how agents handle documents inside a real office setting where things get messy and complicated. The authors are Guo, Zhang, and others from Fudan University and Zhejiang University.

Lu: They are specifically focusing on archive-grounded reasoning; that means the agent has to find evidence only within a fixed collection of documents, not by searching the live web or some open environment.

Meng: So archive-grounded reasoning means the agent is restricted to a specific set of files, which is much harder than just answering questions from general knowledge.

Lalam: And what they are stressing here is agentic exploration; that means the agent has to actively navigate and interact with those files using tools to figure things out.

Tom: So when you put it together, AGORA is trying to create a test that checks if models can do all three things at once: find evidence in archives, explore deliberately, and handle messy workplace data.

Jane: It sets up a very specific scenario where the agent has to reconcile inconsistencies in terminology and units across different files before it can produce an answer. That's a big hurdle for any system trying to reason over unstructured data.

Lu: The paper is aiming to create a benchmark that doesn't just test simple reading comprehension of documents, but how well an agent can actually perform multi-hop reasoning across those documents using tools like bash commands.

Meng: That implies the required skills go beyond just language understanding; they need to be able to execute actions and use tools effectively within a file system context.

Lalam: It’s about moving past benchmarks that just test if a model can find a fact in one document, toward testing if it can manage an entire knowledge base of files.

Tom: So it’s setting the bar high for what we expect from agents that are supposed to be working with real, complex corporate data.

Jane: And the implications are that we need to move beyond simple retrieval systems and toward agents capable of deep, systematic investigation within a document archive.

The paper's summary: Tom: So the paper outlines exactly what AGORA is designed to do—it’s a benchmark pairing three hundred sixty-two questions with eight domain collections of nine thousand six hundred sixty-four documents and 372M tokens <ref:2606.24526#pg1>.

Jane: That scale really puts things into perspective, doesn't it? It’s a massive amount of authentic workplace material that agents have to sift through.

Lu: The key is that this collection is so large, it far exceeds the context window of current models, which forces the agent to explore deliberately rather than just scanning exhaustively.

Meng: So if a model tries to read everything at once, it just fails because there’s simply too much data in its active working memory.

Lalam: The paper describes how they build this using an agentic pipeline that combines cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering to construct those questions <ref:2606.24526#pg1>.

Tom: They are also focusing on how they handle the messy nature of the input by normalizing various file types—PDFs to Markdown, Excel sheets to table profiles—into plain text so a language agent can read them directly.

Jane: That standardization is crucial because in a real workplace, you’re dealing with documents in all these different formats simultaneously. It forces the language agent to handle the translation layer itself.

Lu: The goal of that construction pipeline is to ensure that the generated tasks are grounded in seed chunks and then refined to enforce consistency and naturalness, while obfuscation removes any residual leakage from the original source material <ref:2606.24526#pg3>.

Meng: I’m interested in how they manage that leakage-preventing part because if they aren't careful, the agent might just memorize parts of the documents instead of actually reasoning over them.

Lalam: The paper is really aiming to stress archive-groundedness by making sure the evidence for any given query is truly sparse and scattered across multiple files <ref:2606.24526#pg1>.

Tom: So in short, AGORA is a carefully constructed test environment designed to see if agents can perform complex, cross-domain reasoning over large, messy document archives.

Jane: It’s a very specific test of whether an agent can manage the complexity of real corporate data across different formats and domains. Now we need to look at how they actually build this whole testing suite.

The paper's improvements: Tom: Now that we know what the benchmark is, let’s talk about how the authors suggest they improved things compared to older benchmarks <ref:2606.24526#pg3>.

Jane: They introduced a multi-stage pipeline for building the benchmark itself, starting with cross-document task synthesis.

Lu: That synthesis step is key; it’s not just about pulling questions from existing sources, it’s about progressively enhancing them through refinement to enforce consistency and naturalness.

Meng: And then the quality control stage is where they really ramp up the rigor, including applying difficulty filtering and using a threemodel panel to discard tasks that are too easy.

Lalam: The goal of this whole pipeline is to make sure that when an agent tries to solve a task on AGORA, it’s genuinely struggling with reasoning over the archive rather than just knowing some general facts.

Tom: They also focus heavily on evidence reconciliation, which they define as forcing the agent to handle inconsistent terminology, units, and time conventions before computing an answer <ref:2606.24526#pg1>.

Jane: That reconciliation step is critical because in a real document archive, you often have documents using different standards for the same thing, and the agent has to figure out how to harmonize those differences.

Lu: It sounds like they’re pushing agents toward a more deliberate exploration strategy, where they are required to plan their exploration and reason over evidence selected from the archive instead of just scanning everything <ref:2606.24526#pg1>.

Meng: That idea of planning before execution is something that translates directly into how we design agent architectures—we need the planning capability built in.

Lalam: They also mentioned using two subagents for final task review, one blind and one with a reasoning chain, to determine the reliability of the task before human annotation <ref:2606.24526#pg3>.

Tom: So they're moving beyond just creating a dataset to actually engineering a pipeline that generates difficult, well-vetted problems for agents to tackle.

Jane: It shows they aren't just looking for models that can read documents; they are looking for agents that can perform complex, multi-step cognitive tasks under realistic conditions. Now we have the scale and the process, but what do the actual results look like when we run models on this AGORA setup?

Conclusion: Tom: So to wrap up, AGORA is a very demanding test for agents that need to handle massive document archives and perform deep cross-domain reasoning under real-world constraints.

Jane: The main point is that even the strongest models only reached fifty-nine point four percent accuracy, which suggests that archive-grounded reasoning remains far from solved for current systems.

Lu: It’s a clear signal that true reasoning requires deliberate exploration of the archive rather than just exhaustive scanning, which is a core finding of this work <ref:2606.24526#pg1>.

Meng: From an engineering standpoint, the results show that performance splits into two tiers, and smaller models are not functional on this benchmark at all.

Lalam: The implication is that for agents to succeed in this space, they need to be capable of handling the scale and complexity we’re seeing here.

Tom: So we’re looking forward to seeing how these new requirements shape the next generation of models when we start training them.

Jane: It’s an important piece of work because it clearly shows where the current limitations are, and what needs to be addressed in the next phase of agent development <ref:2606.24526#pg1>.

Lu: The paper concludes that archive-grounded document reasoning remains far from solved for current models.

Meng: I just wonder if this benchmark is robust enough when we try to deploy these agents on real traffic, because they say pretraining corpora might absorb more documents of this kind and erode the guarantee of the closed-book filtering safeguard against knowledge leakage over time.