DocAtlas: Long-Document Understanding as Mutable-State Interaction

arXiv:2608.07527 · cs.CL, cs.AI · Submitted 2026-07-21 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "DocAtlas: Long-Document Understanding as Mutable-State Interaction".

Jane: The paper was written by Hongchen Wei, Yuanzhe Wang, Bei Liu, Yifan Yang, Qi Dai et al. from Wuhan University and Microsoft.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back, everyone. Today we're looking at a paper that's been making waves in the document understanding world. It's called "DocAtlas: Long-Document Understanding as Mutable-State Interaction." Jane, when you first saw that title, what went through your head?

Jane: Honestly, Tom, I had to read it twice. "Mutable-state interaction" sounds like something from a physics textbook, not a paper about reading documents. But once I dug in, it makes perfect sense. It's about treating a document like a living thing that changes as you explore it, rather than a static pile of pages.

Tom: Right, and that's the key shift here. Most systems treat a document as this fixed thing. You retrieve a chunk, you read it, you answer. But DocAtlas says, no, let's let the document state evolve as the agent works through it. Every time the agent takes a note, every time it reads a page, the document's internal map gets updated.

Jane: Exactly. And that's why the title matters. "Mutable" means changeable. So the document isn't just sitting there waiting to be read. It's actively being reshaped by the reading process itself. That's a fundamentally different way of thinking about how AI should handle long documents.

Tom: And it's not just a fancy idea. They've got real results to back it up. We'll get into the numbers later, but let me just say, the improvements are substantial. I mean, they're beating human expert performance on one of the big benchmarks.

Jane: Which is wild when you think about it. We're talking about documents that can be dozens or hundreds of pages long, with charts, tables, figures, all mixed together. And this system is navigating that better than a human expert who's been trained on this exact task.

Tom: So the title is really a promise. It's telling you, this isn't just another retrieval system. This is a system that learns and adapts as it goes. And that's what we're going to unpack over the next few segments.

Jane: And I think the most exciting part is what this means for smaller models. Because the paper shows you can take a relatively small AI model, train it in this environment, and get it performing at a level that's surprisingly close to the big proprietary models. That's a big deal for accessibility.

Tom: Absolutely. But let's not get ahead of ourselves. We've got a lot to cover, and I want to make sure we do this paper justice. So stick around, because next we're going to look at the actual summary and what the authors claim to have achieved.

Summary: Tom: So Jane, we've established that the title is about treating documents as living, changeable things. Now let's talk about what the paper actually claims to have built. The summary is pretty dense, so let's break it down.

Jane: Okay, so the core idea is that they've built a system called DocAtlas that wraps a vision-language model with a set of tools. The agent can search, read, take notes, and review its own notes. But the crucial part is that the document's index, the tree structure that guides search, gets updated as the agent learns things.

Tom: Right, so it's not just about giving the agent tools. It's about making those tools affect the state of the world. When the agent reads a page and writes a note, that note gets written back into the document tree. So the next time the agent searches, it's searching a smarter, more informed index.

Jane: And they've got this really clever design where search and reading are decoupled. Search proposes candidate regions, but the agent decides which pages to actually read. That way, you're not wasting context on pages that the search thought were relevant but actually aren't.

Tom: That's a huge efficiency win. And they've got this note-taking system that's not just a summary. It's structured evidence. Each note has what was found, where it was found, and what modality it came from. Text, table, image, formula. So the agent can go back and verify its own reasoning.

Jane: And the results are pretty stunning. With GPT-five point four, they hit seventy-one point four percent on MMLongBench-Doc, which is a benchmark for long document understanding. The human expert reference is sixty-five point eight percent. So they're beating human experts by almost six points.

Tom: But here's the part that really gets me excited. They took a small model, Qwen3 point 5-4B, which is tiny compared to GPT-five point four, and they trained it with reinforcement learning inside this environment. It went from fifty-four point four percent with direct input to sixty-three point seven percent after training. That's a massive jump for a model that size.

Jane: And that's the real story here. It's not just that a big model can do well with good tools. It's that the environment itself is teachable. The small model learns when to search, when to read, when to take notes. It's not just following a script. It's developing a strategy.

Tom: So the summary is really about two things. First, a better way to interact with long documents. Second, a way to train smaller models to use that interaction effectively. And both of those have big implications for how we build document understanding systems in the future.

Jane: And we're just getting started. Next, we're going to look at the specific improvements they're proposing over existing methods. Because they didn't just invent this from scratch. They're building on a lot of prior work and fixing its weaknesses.

Improvements: Tom: So Jane, we've covered the big picture. Now let's get into the weeds a bit. What are the specific improvements DocAtlas brings over what came before? Because there's been a lot of work on retrieval-augmented generation and agentic document understanding.

Jane: Right, so the paper identifies three main design principles. The first is self-improving retrieval. Most systems use a static index. You embed all the pages once, and that's it. DocAtlas builds a hierarchical tree of the document, and that tree gets enriched as the agent finds evidence.

Tom: So it's like the document's table of contents is learning. When the agent finds something important on page thirty-seven that finding gets attached to the section that covers page thirty-seven. So next time, a search for related information will see that annotation and know where to look.

Jane: Exactly. And the second principle is selective evidence access. In a lot of systems, when search returns a candidate section, the agent reads the whole thing. But if that section spans twenty pages, that's a huge waste of context. DocAtlas separates finding from reading. Search proposes, but the agent decides what to actually consume.

Tom: And that's a big deal for efficiency. The paper shows that raw tree search returns about sixteen point six five pages on average, which is way too many to read directly. But the full reading trajectory only inspects about five point seven nine pages while achieving better evidence coverage. So the decoupling really pays off.

Jane: And the third principle is active working memory. This is the note-taking and review system. The agent writes structured notes with source-attributed evidence, and those notes can be retrieved later. So the agent doesn't have to keep every page it's ever read in its context window. It can compress, archive, and revisit when needed.

Tom: And this is where the reinforcement learning piece comes in. The paper shows that after RL training, the small model's behavior changes. It searches less but reads more. It takes more notes and reviews them more often. The model is learning a strategy, not just following a prompt.

Jane: And that's a really important distinction. A lot of agentic systems just prompt a big model to use tools. But DocAtlas actually trains the policy. The environment is fixed, but the model learns when to invoke each tool. That's a fundamentally different approach.

Tom: And it shows in the numbers. The RL-trained 4B model uses twenty-eight percent fewer search calls but three hundred twenty percent more review calls. It's not just doing more work. It's doing smarter work. It's learning that checking its own notes is more valuable than blindly searching again.

Jane: So the improvements are really about three things. A smarter index that learns, a separation between finding and reading, and a memory system that the agent actively manages. And together, those three things produce results that beat human experts.

Tom: And we've got a lot more to dig into. Next, we're going to look at the actual first page of the paper and see how they frame the problem and what motivates this whole approach.

First Page: Tom: So Jane, we've talked about the improvements. Now let's look at the actual first page of "DocAtlas: Long-Document Understanding as Mutable-State Interaction." How do they frame the problem?

Jane: Well, the first page really sets up the challenge. Real-world documents like financial reports, legal contracts, and scientific papers spread information across dozens or even hundreds of pages. And they combine text with tables, figures, and charts in complex layouts. So answering questions over these documents requires finding and combining evidence that's scattered everywhere.

Tom: And they point out that simply feeding the whole document into a model doesn't work. Even if it fits in the context window, performance drops because irrelevant content competes for attention. That's the "lost in the middle" problem. The model gets distracted by all the noise.

Jane: Right, and they also mention that early retrieval-augmented generation systems, or RAG, have a fundamental limitation. They select a fixed set of pages before generation. So the model has no control over which pages to inspect more carefully or how to revise the search based on partial findings.

Tom: And that's where the agentic approaches come in. Recent work has models interacting with documents through iterative tool use. But the paper points out a key weakness. Most of these systems rely on frozen proprietary backbones, and their behavior is specified through prompting rather than learned from data.

Jane: And that's the gap DocAtlas fills. They want to train the agent itself, not just prompt it. And they specifically mention DocDancer as an example of a system that trains tool use but has a limitation. It uses a text-only LLM as the controller and relies on an external VLM for visual understanding. So the agent learns when to call a tool but not how to interpret visual content itself.

Tom: So the first page is really about identifying the missing piece. We've got retrieval, we've got agentic tool use, but nobody's really trained the whole thing end-to-end in a way that lets the model learn visual understanding and tool use together.

Jane: And that's what DocAtlas does. It exposes the document interface as a mutable environment where the agent policy can be trained. The same interaction protocol is used at inference time and during reinforcement learning. So the model learns by doing, not by imitation.

Tom: And the results on that first page are the hook. With GPT-five point four, they hit seventy-one point four percent on MMLongBench-Doc, beating the human expert reference of sixty-five point eight percent. And the RL-trained Qwen3 point 5-4B reaches sixty-three point seven percent, compared to a fifty-four point four percent direct-input baseline.

Jane: So even on the first page, they're making a strong case. This isn't just a theoretical framework. It's a system that produces measurable, significant improvements. And it works for both huge proprietary models and small open-weight ones.

Tom: And that's what we're going to wrap up with. But before we do, I want to bring in some other voices to get their take on what this means for the field.

Conclusion: Tom: Alright, we've covered a lot of ground on "DocAtlas: Long-Document Understanding as Mutable-State Interaction." Let's bring in Lu, Meng, and Lalam to get their final thoughts before we wrap up.

Jane: Lu, you've been quiet. What's your big-picture take on this?

Lu: I think the most exciting implication is that this changes what we think of as the model's job. DocAtlas isn't just about reading better. It's about teaching a model to manage its own information-seeking process. That's a step toward agents that don't just answer questions but know how to find answers. And the fact that they can train a 4B model to do this well suggests that intelligence isn't just about scale. It's about the right environment and the right training signal.

Meng: From an engineering standpoint, I'm impressed by the practical design. The decoupling of search and read is a real efficiency win. And the note system with archival means you can run these agents without blowing up your context window. That's crucial for real-world deployment where you're dealing with cost and latency constraints. The fact that they show open-weight auxiliaries work almost as well is also a big deal for anyone who doesn't want to depend on a proprietary API.

Lalam: If I may add, the cultural impact here is significant. This system makes long, complex documents more accessible. Think about legal contracts, medical research, government filings. If a small, open-weight model can navigate these effectively, that democratizes access to information. It means organizations without massive compute budgets can still build capable document understanding systems. And that could improve transparency and accountability in fields where documents are gatekeepers.

Tom: That's a beautiful way to put it, Lalam. And Jane, how do you want to send us off?

Jane: I think the takeaway is that DocAtlas shows us a path forward. It's not just about bigger models or longer contexts. It's about building better interfaces between models and information. The mutable state idea, where the document changes as you learn, is genuinely new and it works. We're saying goodbye to this paper, but I have a feeling we'll be seeing its influence for a long time.

Tom: And that's a wrap on "DocAtlas: Long-Document Understanding as Mutable-State Interaction." Thanks to Lu, Meng, and Lalam for joining us. And thanks to all our listeners. We'll be back with another paper soon. Until then, keep reading.

Hongchen Wei, Yuanzhe Wang, Bei Liu, Yifan Yang, Qi Dai, Kai Qiu, Yunsheng Li, Dongdong Chen, Chong Luo, Zhenzhong Chen, Baining Guo

Wuhan University · Microsoft

cs.CL, cs.AI

Submitted: 2026-07-21

Updated: 2026-08-11

Code: https://github.com/VectifyAI/PageIndex

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 69/100

The gist: DocAtlas is a system that treats long-document understanding as a mutable-state information-seeking process.

Key concepts

Mutable-State Interaction
The core idea is treating a document not as a static file but as something that changes. Every time an AI agent reads a page or takes a note, the document's internal map is updated. This evolving state allows the system to adapt and learn as it processes information.
Decoupling Search and Reading
This design separates the process of finding relevant information from actually consuming it. Search suggests candidate regions, but the agent decides which specific pages to read. This prevents wasting context on irrelevant sections, leading to significant efficiency gains.
Active Working Memory
The system uses a structured note-taking method where every piece of evidence is recorded with its source and location (text, table, image). This allows the agent to archive and revisit its own findings without needing to keep every single page in its limited context window.
Reinforcement Learning on Small Models
DocAtlas was trained using reinforcement learning within its interactive environment. This allowed relatively small AI models to develop a strategy—learning when to search, read, or review notes—achieving performance levels previously seen only in much larger models.

Terminology

Summary

DocAtlas is a system that treats long-document understanding as a mutable-state information-seeking process. It is instantiated as a mutable document harness: an external environment that determines what document information is searched, read, stored, reviewed, and shown to the model at each step. Given a document and question, the harness exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence. DocAtlas combines self-improving retrieval, selective evidence access, and active working memory under a fixed context budget. The same harness supports inference-time use with large VLMs and end-to-end reinforcement learning for compact VLM agents.

The paper states: "We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process. We instantiate DocAtlas as a mutable document harness: an external environment that determines what document information is searched, read, stored, reviewed, and shown to the model at each step. Given a document and question, the harness exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence. DocAtlas combines self-improving retrieval, selective evidence access, and active working memory under a fixed context budget. The same harness supports inference-time use with large VLMs and end-to-end reinforcement learning for compact VLM agents."

The method is built around three design principles: self-improving retrieval, selective evidence access, and active working memory. Self-improving retrieval organizes the document as a hierarchical tree that S EARCH can navigate and update during exploration. Selective evidence access separates finding evidence from consuming it: S EARCH identifies relevant sections, and R EAD lets the agent decide which pages to inspect and in which modality. Active working memory is implemented through N OTE and R EVIEW, which record, optionally archive, and review evidence so the agent can reason across many steps under a fixed context budget.

The paper describes the formalization: "Given a document D = d1,..., dN consisting of N pages and a natural-language query q, the agent produces a trajectory τ = (q, u1, o1, u2, o2,..., uT, a), where each action ut for t < T is a tool call (toolt, argt) ∈ T × A, and the trajectory ends with a terminal action uT = finish(a) that outputs the final answer a. For each tool call, the environment returns an observation ot = env(toolt, argt; S (t)), where S (t) = (D, G (t), M(t), H(t)) is the environment state comprising the document D, the document index G (t), the note store M(t), and the set of previously explored pages H(t) ⊆ [1, N]. The tool set T = S EARCH, R EAD, N OTE, R EVIEW has a fixed action space but no fixed execution order."

The hierarchical index is constructed once per document by a VLM in a question-agnostic manner: "the model receives structured markdown extracted from the document together with parsed visual content, including figure/table/chart captions and layout-derived cues, and outputs a JSON tree with section titles, page ranges, and summaries. This preprocessing does not use benchmark questions, answers, or evidence annotations."

S EARCH takes a search query and the current index state, and an auxiliary LLM selects tree nodes by using the hierarchy, preferring unvisited regions when useful, and using annotations from prior exploration. When the agent records findings via N OTE, the evidence entries are propagated back to the tree: "For each cited page p, the finest-grained covering node v ∗ = arg minv: p∈[pvs,pve] pve − pvs receives a compact annotation in f indingsv∗. The index thus evolves: G (t+1) = enrich G (t), evidencet, so later S EARCH calls see accumulated findings alongside original summaries."

R EAD decouples retrieval from reading: S EARCH proposes candidate regions, while R EAD gives the agent explicit control over the final evidence set P ⊆ [1, N]. Retrieval results are therefore proposals rather than mandatory context. R EAD returns, for each page p, three content types: a full-page layout image ximg p, a structured markdown rendering xmd p (extracted by MinerU), and a set of cropped sub-images xsub p = xsub p,1,... for figures and charts.

N OTE records findings as a structured tuple n = (found, evidence, plan), where found summarizes what has been established, evidence = (ej, pj, typej) Jj=1 is an array of typed, source-attributed entries (typej ∈ text, table, image, formula), and plan states remaining information gaps. N OTE triggers two side effects: "When archival is enabled, prior R EAD outputs in ht can be replaced in-place with short placeholders, yielding ĥt with ĥt ≪ ht. Evidence entries are also written back to the tree, closing the feedback loop with S EARCH. Notes are constrained to be extractive and page-grounded: each evidence entry must be copied or normalized from a previously read page and paired with its source page and modality."

R EVIEW reviews the note store and returns only findings relevant to a query: Each note ni ∈ M(t) is projected to a compact note card ci = (idi, stepi, foundi, pagesi) that strips the full evidence payload. An auxiliary LLM receives c1,..., cM(t) and selects the relevant subset.

For RL training, the paper states: "We optimize a single VLM policy over complete tool trajectories using outcome rewards. The environment components, including the tree, tool executors, and auxiliary subroutines used by S EARCH and R EVIEW, are frozen; the policy learns when to search, what to read, what to record, and when to answer. The reward is computed from the final answer only: If the last visible answer lacks a parsable field, the reward is zero; otherwise, we extract the boxed content and apply the LongDocURL type-aware answer score: r = ScoreLongDocURL ExtractBoxed(a), a⋆. The optimization uses GRPO with DAPO-style asymmetric clipping (ϵlow =0.1, ϵhigh =0.3), n=8 rollouts, group-normalized advantages, and token-level KL regularization (λKL =0.01)."

The main results show: "With GPT-5.4, DocAtlas reaches 71.4% on MMLongBench-Doc, exceeding the human-expert reference of 65.8%. A Qwen3.5-4B VLM trained with end-to-end RL in the DocAtlas environment reaches 63.7%, compared with a 54.4% direct-input baseline. Specifically, GPT-5.4 rises from 62.4 direct accuracy on MMLongBench-Doc to 71.4 inside DocAtlas, a 9.0-point gain over the 65.8 human-expert reference. The paper also reports: GPT-5.4 improves from 55.1 to 75.6 LasJ on FinRAGBench-V and from 66.9 to 78.8 on LongDocURL; GPT-5.2 also reaches 70.6 on MMLongBench-Doc and 77.5 on LongDocURL."

For compact policies: Qwen3.5-4B improves from 54.4 with direct input to 61.0 with DocAtlas, and Qwen3.5-9B improves from 58.0 to 61.6. After RL: "On MMLongBench-Doc, Qwen3.5-4B moves from 61.0 to 63.7 after RL, and Qwen3.5-9B from 61.6 to 64.4. The gains transfer beyond the RL data source: FinRAGBench-V improves from 67.9 to 71.7 for 4B and from 69.8 to 72.6 for 9B."

The ablation study shows: The largest losses come from visual access. Removing layout images lowers ALL from 71.4 to 65.7, with large drops on figures and charts; removing cropped sub-images lowers ALL to 67.2. Also: Without R EAD, the agent consumes S EARCH candidates directly and ALL falls to 67.2. The mutable-state components each contribute consistent gains: removing the evidence field, R EVIEW, N OTE, plan field, or tree annotation yields 67.8–68.3 ALL.

Tool-use analysis shows RL changes the compact policy's behavior: Total calls grow by only 10.5%, but the mix shifts sharply: S EARCH decreases by 28.0%, while R EAD, N OTE, and R EVIEW increase by 28.3%, 66.7%, and 320.0%. The clearest change is R EVIEW: the no-RL 4B policy almost never uses it (0.05 calls/question), whereas the RL policy reaches 0.21, matching GPT-5.4's 0.20 rate.

Evidence efficiency analysis shows: "Raw tree search has high coverage (87.45 All-Hit) but is intentionally broad, returning 16.65 pages on average... The full reading trajectory inspects far fewer pages (5.79) while achieving the best page-level F1 (58.99). Tree annotations matter: removing them drops raw-search All-Hit from 87.45 to 80.05 and page F1 from 38.33 to 29.40."

Auxiliary robustness analysis shows: Qwen3.5-35B-A3B loses only 0.3–0.4 points, and 'Self' keeps both policies within 2 points... The fully open-weight self setting still reaches 62.0/63.2 for the 4B/9B policies.

The paper concludes: "DocAtlas treats long-document understanding as mutable-state interaction: agents search a visual-aware tree, read pages, write grounded notes, and review prior findings. This turns a long document into state that the policy updates while it works. Across three benchmarks, DocAtlas improves large VLM agents and helps compact Qwen policies learn tool use through RL. The gains come from multimodal reading, decoupled S EARCH/R EAD, mutable notes, tree annotations, and S EARCH/R EVIEW auxiliaries that can be replaced by open-weight or self models. Overall, long-document understanding depends not only on context length, but also on how the interface lets a model find, remember, and use evidence."

Improvements for AI systems

Based on the DocAtlas paper, here are the specific improvements I can implement in AI systems and the resulting capabilities:

Improvement: Replace static retrieval-augmented generation (RAG) with a mutable-state interaction loop where the document index, note store, and explored-page set are updated by the agent's own actions during an episode.

What the improved system can do:

  • Search a hierarchical tree index that evolves as evidence is gathered (self-improving retrieval)

  • Decouple search from reading: search proposes candidate regions; the agent decides which pages to actually consume

  • Write source-attributed, page-grounded notes that feed back into the tree for later searches

  • Review prior notes when combining evidence across distant pages

  • Operate under a fixed context budget by archiving bulky read observations into compact placeholders

Overall capability improvement: The improved AI system can navigate, search, read, and reason over documents spanning dozens to hundreds of pages with complex layouts, tables, figures, and charts—achieving human-expert-level performance with large VLMs and near-competitive performance with compact open-weight models trained via RL, all while maintaining a fixed context budget and providing source-grounded, verifiable answers.

Sources

Related papers