DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding

arXiv:2608.07067 · cs.AI, cs.CL, cs.IR, cs.MM · Submitted 2026-08-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding".

Jane: The paper was written by Hanshu Yao, Jianfeng Zhong, Niu Lian and Jinpeng Wang from Harbin Institute of Technology, Shenzhen and Tsinghua Shenzhen International Graduate School, Tsinghua University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We just spent the morning on a paper that made me want to reorganize my own desk. It tackles a nasty problem: answering questions from documents that run hundreds of pages, where the answer might hide in a table on page 87. Most systems grab a fixed pile of pages up front and then pray. This one keeps hunting.

Jane: And that hunting is the whole trick. The authors — Hanshu Yao, Jianfeng Zhong, Niu Lian, and Jinpeng Wang — call it memory-guided evidence discovery. Instead of one shot, the system goes round after round. It remembers which pages looked useful, which did not, and where the question has already taken it. Like a detective with a notepad.

Lu: The notepad has three layers. Document Schema Memory stores the stable structure: the document type and the section map. Page Belief Memory tracks how likely each page is relevant, and it shifts with feedback. Question Episodic Memory holds the clue trail for the current query. Stable knowledge, dynamic relevance, and local experience — the paper separates all three.

Tom: And they put real math underneath?

Lu: Real math. Every page carries a Beta distribution. Reasoning feedback updates those distributions through Bayesian belief updating. Thompson sampling then decides which pages deserve another look, balancing safe bets with uncertain leads.

Meng: It pays off on the leaderboards. MMLongBench-Doc lands at 71.3 percent, LongDocURL at 81.1, PaperTab at 80.4. Against the strongest agentic baselines, that is a 15-point jump on PaperTab.

Jane: Fifteen points on the table-heavy benchmark, which makes sense. The system zooms into high-resolution crops of tables and charts when the page-level view gets too blurry. They call it adaptive-granularity evidence access.

Lalam: Stepping back, we keep shoving documents into context windows and calling that reading. This paper treats reading as search with memory and feedback. It also does it cheaper — about 0.41 times the retrieval cost of a strong baseline for higher accuracy. Efficiency and accuracy moving together is rare, and that is why this matters beyond the benchmark tables.

Meng: The gains are not just averages either. Unanswerable cases jump to 78.8 percent, which is where most systems collapse into hallucination.

Tom: The code is already public, too, so this is not vaporware. Page one sets up the problem with a sharp critique of everyone else's approach.

Page 1 of the paper: Jane: We just sketched the headline results. Now page one explains why existing systems fail, and the opening argument lands hard. The central challenge is dynamic: locating, updating, and integrating relevant pages under a limited evidence budget. Passively reading a long input does not cut it.

Lu: Context budgets are the wall. A hundred-page PDF will never fit comfortably in a model window, so you have to choose pages. The paper's critique is that most systems choose once, then live with the consequences. Single-turn retrieval fixes a candidate set at the start, and if a key chart is missed in that first pass, you are done.

Meng: Then there are iterative methods that add more rounds. The paper credits them for trying but says they drop the ball on memory. Cross-round information is mostly preserved by rebuilding the context each time. That is closer to repeated independent retrieval than to actual evidence exploration.

Tom: That is the gap this system is built for. Retrieval that can recover from early mistakes. The authors frame the whole thing as dynamic evidence exploration, not passive reading.

Lalam: And the framing borrows from cognitive science. Complementary learning systems — the brain separates slow, stable knowledge in the neocortex from fast, flexible traces in the hippocampus. The paper takes that separation and builds it straight into the architecture.

Jane: One more thing from the intro: the evidence itself is messy. The answer can live in text, tables, figures, or visually structured layouts. A retriever that only searches keywords will starve.

Meng: Their phrase stuck with me — "stateful exploration." The model should carry a state of where it has been and what it half-learned, instead of starting from zero every round.

Tom: So the intro lines up three failure modes against the three memory layers. Static retrieval is rigid. Iterative retrieval forgets. The bet here is that structured memory fixes both. Then Figure 1 draws that contrast.

Jane: And that figure is not decorative — it is the entire thesis in one image. We should unpack it.

Page 2 of the paper: Lu: We left off at that thesis-in-one-image. Page two opens with the figure, and it draws three retrieval paradigms side by side. Single-turn retrieval is a fixed candidate pool feeding a one-shot pipeline — rigid, brittle. Iterative retrieval stacks context flat, like piling papers on a desk without sorting them. The proposed framework separates memory from evidence discovery and updates both continuously.

Tom: The figure even labels the memory layers with brain names. Prefrontal cortex for slow semantic memory, hippocampus for fast episodic memory. It is a metaphor, sure, but it guides the engineering.

Jane: And the captions tell a story on their own. "Rigid and One-Shot Pipeline" for the old way, "Flat Context Accumulation" for the iterative way. The labels alone show where the field went wrong.

Meng: Then come the three contributions. A tri-level memory framework. Bayesian page belief updating with Thompson sampling and spatial contiguity propagation. Experiments on three benchmarks showing consistent gains. Short list, heavy claims.

Lu: Related work starts right after. Static retrieval-augmented approaches pick evidence first and reason later. Some add visual retrievers, structured retrieval, or fine-grained localization, but the selection happens before reasoning begins. The paper's verdict: once key evidence is missed, later reasoning has little chance to recover it.

Jane: The second bucket is iterative retrieval, and the paper singles out SimpleDoc. SimpleDoc lets an agent keep retrieving when evidence is insufficient. But it lacks a mechanism for transferring states across rounds. The paper says that process is closer to repeating the same search than building on it.

Tom: So the literature review sets up a clean dichotomy. Static methods cannot adapt. Iterative methods do not remember. The method section starts on page three, with the memory definitions.

Lalam: What strikes me is the vocabulary. Document Schema, Page Belief, Question Episodic — that maps onto long-term structure, working state, and short-term experience. It gives researchers a language for what previous systems kept implicit.

Jane: And the phrase in the figure — "decouple memory and dynamic evidence discovery" — that might be the one-line summary of the entire paper.

Tom: It is. Page three then shows the schema layer getting built before any question arrives.

Page 3 of the paper: Meng: The figure gave us the map, and now page three fills in the details. It finishes the related work with agent memory, and that subsection is basically a complaint. LLM agents have explored hierarchical storage, long-term memory updating, associative organization. But most of that targets open-ended interaction or video-style experiences unfolding over time. It does not handle documents where text, figures, tables, and layout cues are deeply intertwined.

Jane: Exactly. A PDF is not a video. The signals are heterogeneous and they sit on the same page, physically entangled. The proposed tri-level memory is positioned as the missing piece for that mess.

Lu: Then the method section formalizes the setup. Document D with N pages, query q, up to T retrieval-reasoning rounds. Offline, page embeddings and summaries get computed once. Online, the memory state is written as a triple: schema, belief, episodic.

Tom: The first piece is Document Schema Memory, built offline and fixed during inference. Every page gets a summary, and the system aggregates them into one package: the document type, a structural index of page ranges with topic labels, and a global summary. That package is the navigation map.

Meng: So the schema is a map of the building, drawn before anyone asks a question. It says a survey report runs pages one through ten on demographics, then twenty to thirty on opinions. That map never changes, no matter who is asking.

Jane: And because it is query-independent, the cost gets amortized. Build it once, reuse it across every question on that document. That is a smart place to spend offline compute.

Tom: It also gives the retrieval agent a navigation prior. When a later query mentions unemployment, the schema can point toward the economy section before the visual retriever even runs.

Lalam: Persistent structure on one track, dynamic beliefs on another. That separation keeps the offline investment useful and the online state lean. The second memory layer is where the probability enters, and that is page four.

Page 4 of the paper: Lu: Page four is where the math arrives. Page Belief Memory gives every page a Beta distribution — two numbers that encode how strongly evidence supports the page being relevant versus irrelevant. The posterior mean becomes the page's accumulated relevance confidence.

Tom: So each page is basically a probability coin being flipped during retrieval?

Lu: Not flipped — sampled. The system draws from that distribution when choosing pages. That is the Thompson sampling bit, and it is coming on page five.

Meng: The initial beliefs come from the visual retriever, ColQwen2.5. It computes token-level similarity between the query and each page using a late-interaction mechanism, normalizes the score, and converts it into a Beta prior. A strength parameter S controls how much the visual signal is trusted — they set it to five.

Jane: But the elegant part is the update rule. After each reasoning round, the model reports useful pages and irrelevant pages. Useful feedback increments the alpha parameter; irrelevant feedback increments beta. Beta-Bernoulli conjugacy keeps the update a simple addition.

Tom: Then the spatial propagation step, and I want to underline this. The paper notices that evidence in long documents clusters locally. If page 20 is valuable, pages 18 through 22 deserve another look. So positive feedback spreads to neighbors within a small radius, with a decay factor.

Jane: Negative feedback does not spread. That asymmetry stops one bad page from poisoning the whole neighborhood. With a radius of two and a decay of 0.5, close pages get a real boost while far ones fade out.

Lu: The propagation turns document layout into a soft prior. Pages sit next to each other in physical space, and the model exploits that ordering.

Meng: And the paper grounds it in cognitive load theory — the spatial contiguity principle. Related information belongs close together, so the retrieval rule borrows from how humans learn.

Jane: By the end of page four, beliefs have been born from visual scores, updated by reasoning feedback, and spread across neighborhoods. Page five shows how those beliefs steer the next retrieval round.

Page 5 of the paper: Meng: Page five explains the retrieval loop. For each page, the system samples a relevance estimate from its Beta distribution. That sample is Thompson sampling, a classic bandit strategy that balances exploiting confident choices with exploring uncertain ones.

Jane: The sampled value blends with a fresh visual similarity score against the refined query. The blend weight lambda grows over rounds — the schedule is zero, 0.3, 0.6, 0.6. Early on, pure vision; later, accumulated belief takes the wheel.

Tom: After scoring, a candidate pool is built, and a language model reranks it using page summaries plus the schema and episodic memories. The refined set merges into an accumulated evidence set. High-confidence historical pages stay in the conversation.

Lu: Then the reasoner reads the evidence and makes a three-way decision. It answers when evidence is sufficient. It returns not answerable when the document lacks support. Otherwise, it writes a refined query and a note, both stored into Question Episodic Memory. The loop repeats until an answer, a refusal, or the round limit.

Meng: That explicit unanswerable path deserves a pause. The system can say "this document does not contain it." The experiments show strong gains on those questions, where static retrievers tend to hallucinate instead.

Jane: The last mechanism on page five is adaptive-granularity evidence access. When a page is dense, like a table-heavy page, the system appends high-resolution crops to the full-page image. Full-page rendering loses the fine details; the crops bring them back.

Tom: And the reasoner is warned not to request the same pages or elements again. The episodic memory records what has been ruled out, so later rounds keep narrowing the search instead of looping.

Lalam: What stands out to me is that uncertainty is treated as a resource. Sampling gives the system a budget for exploring pages it is not sure about. That is the difference between ranking and deciding.

Jane: One more nuance — lambda starts at zero, so the first round is pure visual retrieval. The beliefs only earn their weight after the reasoner has something to say.

Tom: Conservative design, and smart. Trust the vision first, trust the memory after it has proven itself. Page six switches to benchmarks and experimental setup.

Page 6 of the paper: Lu: Page six lays out the experiments. MMLongBench-Doc has 1,082 questions across 135 documents, averaging 47.5 pages, with some reaching 112. Those questions deliberately mix text, images, tables, charts, layout understanding, and unanswerable cases.

Jane: LongDocURL is bigger — 2,325 question-answer pairs across 396 PDFs, demanding long-document understanding, numerical reasoning, and cross-element grounding. PaperTab offers 393 questions over 307 scientific papers, focused on tables. Three benchmarks, three distinct flavors of pain.

Tom: Evaluation uses GPT-4.1 as an automatic judge, scoring each prediction as correct or incorrect. But two extra metrics matter: Evidence Recall, the share of ground-truth evidence pages the system found, and All-Hit Rate, the fraction of questions where every annotated page was retrieved. That separates retrieval quality from answer luck.

Meng: The backbone is Qwen3.5-VL-9B, an open multimodal model, served with vLLM. ColQwen2.5 encodes pages offline, and MinerU extracts the table and figure crops. The hyperparameters fall into place: prior strength five, the lambda schedule we just mentioned, propagation radius two, decay factor 0.5.

Lu: Table 1 gives the MMLongBench breakdown. The system hits 71.3 overall, with 73.3 on tables and 78.8 on unanswerable questions. The table gain tracks the adaptive-granularity crops. The unanswerable gain tracks the accumulated memory state.

Jane: The competition is serious — GPT-4o, Claude-4-Sonnet, Gemini models, open MLLMs like InternVL3, agentic systems like SimpleDoc and DocLens. The proposed framework tops the entire table.

Lalam: One detail I appreciate: the evaluation protocol matches recent baselines, so the comparison is fair. The paper also validates the judge against human raters later, with 96.7 percent agreement. That diligence makes the numbers credible.

Tom: And the benchmarks were chosen to cover different evidence types — dense tables, long PDFs, scientific papers. That breadth is what makes the average score meaningful.

Meng: Also note the unanswerable category. Many systems collapse there because they refuse to say "not answerable."

Jane: And the paper treats that refusal as a first-class output, not a failure. Page seven asks the harder question: does it still win when everyone gets the same backbone and budget?

Page 7 of the paper: Jane: Page seven starts with a fairness check. They lock the backbone, retriever, page budget, and round count, then compare against SimpleDoc and MoLoRAG. The proposed system still wins — 61.7 over SimpleDoc's 60.1 with the smaller backbone, and 71.3 over 69.3 with the larger one.

Tom: The cross-benchmark table follows. The average across MMLongBench, LongDocURL, and PaperTab lands at 77.6. The strongest agentic baseline sits at 66.1. That is more than an eleven-point gap.

Lu: The ablations reveal where the credit goes. Removing Page Belief Memory drops accuracy from 71.28 to 68.80. Removing all memory modules drops it to 68.47. Removing Bayesian updating entirely lands at 68.80 again. The dynamic belief machinery is doing heavy lifting.

Meng: The retrieval metrics tell the same story. Evidence recall climbs from 28.32 percent in round one to 69.56 percent by round three. All-hit rate jumps from 12.9 to 58.05. The second iteration delivers the biggest leap, which shows the reasoning feedback from round one actually redirects the search.

Jane: Efficiency is the surprise punchline. SimpleDoc always runs three iterations; this system averages 1.24. That is 0.41 times the retrieval cost for higher accuracy. The paper packages it as a 2.4 times efficiency gain.

Tom: And they verified the judge too — 96.7 percent agreement with human raters, Cohen's kappa 0.92. The accuracy numbers are not an artifact of the evaluator.

Lu: The appendix also tests Thompson sampling head-to-head. Replace it with greedy selection and accuracy falls from 71.28 to 68.62. Uncertainty-based exploration carries the whole system.

Meng: Notice the ablation pattern. Removing one module costs a few points, removing all of them costs more. The pieces compound instead of overlapping.

Jane: So memory helps, exploration helps, and the gains survive controlled comparisons. Page eight then shows the whole system on a single concrete question, and that example is a lot of fun.

Page 8 of the paper: Meng: Page eight gives us a worked example, and it reads like a detective story. The query asks which country's youth show the greatest concern about unemployment. The document is the Arab Youth Survey 2014, 45 pages. The answer hides in a chart.

Jane: Round one retrieves pages 16 through 20. The reasoner reads them and concludes they hold general survey statistics but no per-country breakdown. The chart must be somewhere else. So it writes a refined query mentioning a stacked bar chart and updates its episodic note.

Tom: Round two widens the net — pages 43, 17,

Conclusion: Tom: So the big idea from this paper is that long-document reading works better when you treat it as a hunt with a memory, not a one-time glance.

Jane: Exactly. Three kinds of memory — the document's map, the page-by-page confidence, and the trail of clues you've already found.

Tom: And every round of reasoning writes back into that memory. The system gets smarter about where to look next.

Jane: That's what separates it from the old iterative methods. They just kept pulling pages; DocMemo actually learns from what it saw.

Tom: Those numbers on tables and unanswerable questions really sold me. Seventy-three and a half on tables, almost seventy-nine on unanswerable.

Jane: And it does it with fewer retrieval rounds than the baseline. More accuracy, less work. That's the rare combo.

Tom: The Bayesian belief updating felt like the hidden engine. Every page carries a little probability coin, and Thompson sampling flips it to decide where to explore.

Jane: Smart design. You don't just grab the top pages, you give the uncertain ones a chance to prove themselves.

Tom: The worked example with the Arab Youth Survey was a nice way to close. You could practically watch the system zero in on page twenty.

Jane: It looked like a detective correcting their own assumptions. First guess wrong, adjust, find the chart, done.

Tom: And the code is public, so anyone can try it. That lowers the barrier for the next round of work.

Jane: I hope someone pushes this toward even longer documents — maybe book-length reasoning, or multi-document comparisons.

Tom: Or applies the same memory structure to video understanding. That's a natural next step.

Jane: Good thought. But our time on this one is up.

Tom: We'll be back with something fresh. Until then, keep searching with a notepad.

Jane: See you next episode.

Hanshu Yao, Janfeng Zhong, Niu Lian, Jinpeng Wang

Harbin Institute of Technology, Shenzhen · Tsinghua Shenzhen International Graduate School, Tsinghua University

cs.AI, cs.CL, cs.IR, cs.MM

Submitted: 2026-08-07

Updated: 2026-08-10

Comments: DocMemo is a memory-guided framework for long-document reasoning that uses tri-level memory and dynamic Bayesian belief updating to overcome static retrieval limits and improve evidence tracking. 16 pages, 4 figures, 14 tables

Code: https://github.com/Harrygof/DocMemo

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 74/100

Key concepts

Memory-guided evidence discovery
A retrieval approach where the system iteratively searches through document pages, using memory to track which pages were useful, which were not, and where the query has led. This allows it to recover from early mistakes, unlike static retrieval that fixes a candidate set upfront.
Page Belief Memory
A memory layer that assigns each page a Beta distribution representing its relevance probability. The distribution is updated via Bayesian belief updating based on reasoning feedback, and Thompson sampling is used to select pages, balancing exploitation of confident choices with exploration of uncertain ones.
Adaptive-granularity evidence access
A technique that, when a page is dense (e.g., tables or charts), appends high-resolution crops of specific elements to the full-page image. This preserves fine details that might be lost in full-page rendering, improving evidence extraction for table-heavy documents.

Terminology

Summary

Code Repository: https://github.com/Harrygof/DocMemo

The paper addresses the challenge of long-document visual question answering (DocVQA), which requires models to locate dispersed and heterogeneous evidence across documents spanning tens or even hundreds of pages. Such evidence may appear in text, tables, figures, and visually structured layouts. The authors note that "since document length far exceeds the context budget of existing models, the central challenge of long-document DocVQA is not passive reading of long inputs, but dynamically locating, updating, and integrating relevant pages under a limited evidence budget."

The paper identifies critical limitations in existing approaches:

  • Single-turn static retrieval methods typically fix a candidate page set before reasoning; once key evidence is missed in the initial retrieval stage, subsequent reasoning has little opportunity to recover it. These methods commit to a fixed top-k page set at the outset and struggle to recover from early retrieval errors.

  • Iterative methods allow retrieval over multiple rounds, but they lack cross-round state propagation mechanisms. The paper states that cross-round information is mainly maintained through context reconstruction, making the process closer to repeated independent retrieval. Thus, although retrieval is formally extended to multiple rounds, it still lacks a structured exploration memory that can be continuously updated.

DocMemo is described as a memory-guided framework that formulates long-document reasoning as dynamic evidence exploration. The framework is explicitly motivated by the complementary learning principle of separating stable and rapidly changing information, drawing on cognitive science concepts.

DocMemo maintains a tri-level retrieval state:

  1. Document Schema Memory (M schema): Models query-independent document structural priors. It is constructed offline by generating summaries for each page and aggregating them into a formal structure M schema = (τ, Σ, σ), where "τ denotes the document type and topic description, Σ = (a j, b j, l j) is a structural index consisting of page ranges [a j, b j] and topic labels l j, and σ is a document-level global summary. This memory remains fixed during inference and serves as a query-independent navigation prior."

  2. Page Belief Memory (M belief): Maintains the relevance confidence of each page under the current query. For each page p i, DocMemo uses a Beta distribution to model uncertainty over its latent relevance: M belief = Beta(α i, β i). Here, α i and β i accumulate evidence supporting the page being relevant and irrelevant, respectively. Initialization uses visual relevance computed via the late-interaction mechanism of ColQwen2.5, with a score c i ∈ [0,1] used to set the prior: α i(0) = S·c i + 1 and β i(0) = S(1−c i) + 1, where S controls the strength of the visual prior.

  3. Question Episodic Memory (M epi): Records the episodic reasoning trajectory gradually formed under the current query, including discovered evidence, unmet information needs, and query rewriting history. It is updated only within a single QA session: M epi = (n k, q̂ k), where n k is the intermediate evidence summary/gap analysis and q̂ k is the refined query at round k.

DocMemo treats Page Belief Memory as a retrieval memory state that evolves with reasoning feedback. After each reasoning round, the reasoner outputs useful pages (U t) and irrelevant pages (V t). Based on Beta-Bernoulli conjugacy, beliefs are updated:

  • α i ← α i + I[i ∈ U t]

  • β i ← β i + I[i ∈ V t]

The posterior mean μ i = α i / (α i + β i) represents its accumulated relevance confidence at the current round.

Spatial proximity propagation exploits the locality of evidence distribution in long documents: for i ∈ U t, if 0 < j − i ≤ r, the update is α j ← α j + γ j−i, where r is the propagation radius and γ is the distance decay factor. Negative feedback is not propagated avoiding the suppression of potentially relevant nearby pages. This mechanism transforms local continuity in document structure into a soft prior for page exploration.

The updated Page Belief controls page selection in the next round. DocMemo:

  • Samples a relevance estimate from each page posterior via Thompson sampling: θ i(t) Beta(α i(t), β i(t))

  • Recomputes visual similarity c i between each page and the current query

  • Combines these signals into a candidate score: score i(t) = (1−λ t)c i(t) + λ t θ i(t)

"Here, λ t controls the relative weight between current-query similarity and historical page belief. As iteration proceeds, retrieval gradually shifts from relying only on visual similarity toward incorporating accumulated page confidence. Thompson sampling provides an exploration–exploitation trade-off: high-confidence pages can be repeatedly exploited, while uncertain but potentially relevant pages still retain the opportunity to be explored."

The system selects a candidate page set, uses an LLM reranker with page summaries plus memory context to produce a refined set, and merges into a cross-round accumulated set. The reasoner outputs either a final answer, a not answerable judgment, or a refined query triggering the next round. The loop terminates when the system produces an answer, returns not answerable, or reaches the maximum number of rounds.

Adaptive-granularity evidence access supplements page-level evidence: "For pages containing locally dense information, the system supplements the full-page context with finer-grained visual regions, allowing the reasoner to use both page-level semantics and high-resolution local evidence. This design mitigates the loss of details caused by full-page representation, especially for table-intensive documents."

DocMemo is evaluated on three benchmarks:

  • MMLongBench-Doc: 1,082 questions over 135 long documents, with documents spanning up to 112 pages and averaging 47.5 pages, covering text, images, tables, charts, layout understanding, and unanswerable cases (UNA)

  • LongDocURL: 2,325 question-answer pairs over 396 PDF documents

  • PaperTab: 393 questions over 307 scientific papers focusing on tabular information

The backbone is Qwen3.5-VL-9B, with ColQwen2.5-v0.2 as the visual retriever and GPT-4.1 as the automatic binary judge. Key hyperparameters include prior strength S=5.0, Thompson sampling schedule λ=[0.0, 0.3, 0.6, 0.6], spatial propagation radius r=2, and decay factor γ=0.5.

DocMemo achieves 71.3% on MMLongBench-Doc, 81.1% on LongDocURL, and 80.4% on PaperTab (77.6% average). Relative to the strongest agentic baselines, it improves by 3.7 points on MMLongBench-Doc, 8.8 points on LongDocURL, and 15.0 points on PaperTab. On MMLongBench-Doc's fine-grained categories, DocMemo is especially strong on TAB (73.3%) and UNA (78.8%). Under identical configurations (same backbone, retriever, budget, rounds), DocMemo outperforms SimpleDoc (61.7 vs. 60.1 with Qwen3-VL-8B; 71.3 vs. 69.3 with Qwen3.5-9B).

Removing tri-level memory or Bayesian updating causes the overall accuracy to drop from 71.3% to 68.5% and 68.8%, respectively. The degradation is particularly evident on major categories such as CHA and TXT. Removing adaptive-granularity causes the largest degradation on TAB. Replacing Thompson sampling with greedy selection lowers performance from 71.28 to 68.62, showing the exploration-exploitation balance it provides is necessary for multi-round evidence discovery. Additional ablations on LongURL and PaperTab confirm consistent gains from each component.

DocMemo's cumulative Evidence Recall improves from 28.32% (round 1) to 69.56% (round 3), and All-Hit Rate from 12.90% to 58.05%. DocMemo requires only 0.41× of SimpleDoc's retrieval iterations while achieving 1.18× higher accuracy, yielding a 2.40× overall efficiency improvement. On average, DocMemo uses 1.24 rounds per question (vs. SimpleDoc's fixed 3), and each VLM call processes an average of 5.2 images and 11.5K input tokens with 20.41s average latency.

A case study on the Arab Youth Survey illustrates how DocMemo iteratively discovers that Egypt shows the greatest concern about unemployment, recovering from initially retrieved survey-statistics pages to locate the correct chart on page 20. The authors note: "the iterative process in DocMemo is not merely about increasing the number of retrieval rounds; rather, it continuously reshapes the search space through structured memory, enabling the system to recover from insufficient early evidence."

The paper's stated contributions are:

  1. DocMemo, a memory-guided framework for multi-modal document understanding, which organizes document exploration with a tri-level memory separating document priors, page beliefs, and query-specific reasoning traces

  2. Bayesian page belief updating with Thompson sampling and spatial contiguity propagation, enabling uncertainty-aware and locality-aware dynamic retrieval across rounds

  3. State-of-the-art results on three long-document DocVQA benchmarks with ablations validating the core components

The authors conclude that explicitly modeling and continuously updating document exploration states is key to improving long-document reasoning.

Improvements for AI systems

Here are concrete improvements to AI systems derived from DocMemo, and what the improved system can do:

1. Add separable memory states for long-horizon reasoning.

Maintain three distinct memories: (a) a static document schema (type, topic index, page ranges, global summary) for query-independent navigation; (b) a per-page belief state (Beta distribution over relevance) that accumulates evidence across rounds; (c) an episodic trace of the current reasoning path (found evidence, unmet needs, rewritten queries). The improved system can navigate documents by structure, track uncertainty page-by-page, and remember why it searched where it did—so it doesn't repeat failed retrieval or lose the thread across reasoning steps.

2. Use Bayesian belief updating with spatial proximity propagation.

When a page is confirmed relevant, increment its belief; when irrelevant, decrement it; also partially strengthen neighboring pages (e.g., radius-2, decay 0.5) because evidence in long documents is often locally clustered. Do not propagate negative feedback, avoiding over-suppression of nearby relevant content. The improved system can exploit document locality to discover evidence clusters, recover from pages that were initially missed, and avoid wasting budget re-reading irrelevant neighborhoods.

3. Replace fixed top-k retrieval with Thompson-sampling-based exploration/exploitation.

At each round, sample a relevance score from each page's posterior (not just the mean), combine it with current-query visual similarity using an increasing weight schedule (e.g., λ = 0 → 0.3 → 0.6), and select pages accordingly. This gives uncertain pages a chance to be explored while reliably relevant pages are re-exploited. The improved system can balance between following the current query and following accumulated historical knowledge, and can escape early retrieval errors instead of being stuck with a fixed candidate set.

4. Add adaptive-granularity evidence access for dense local information.

When a selected page contains table-heavy or locally dense content, supplement full-page context with finer-grained visual regions. The improved system can answer table/chart questions accurately without losing high-resolution details that full-page downscaling destroys.

5. Use dynamic stopping and cross-round merging of evidence.

Instead of running a fixed number of retrieval/reasoning rounds, let the reasoner terminate early when it has enough evidence or when it determines the question is unanswerable. Merge page sets across rounds so previously found evidence is always available to the final reasoning step. The improved system can answer with fewer iterations (∼1.24 rounds vs. fixed 3), achieve higher accuracy, and correctly return “not answerable” rather than hallucinating from incomplete retrieval.

6. Guide every step with the accumulated reasoning context.

Feed the current episodic memory (intermediate summaries and gap analysis) into the retriever, reranker, and reasoner so each round is informed by what has already been learned and what is still missing. The improved system performs genuinely multi-round evidence discovery rather than repeated independent retrievals—capable of recovering from insufficient early evidence (e.g., finding a specific chart on page 20 only after initially retrieving statistics pages).

Concrete capability summary of the improved AI system:

  • Handles documents of 100+ pages by dynamically exploring, not passively reading.

  • Recovers from early retrieval misses and improves recall across rounds (e.g., evidence recall from 28% to 70%, all-hit rate from 13% to 58%).

  • Achieves state-of-the-art results on long-document VQA benchmarks: 71.3% on MMLongBench-Doc, 81.1% on LongDocURL, 80.4% on PaperTab.

  • Efficient: uses only 41% of baseline retrieval iterations while improving accuracy 1.18×, giving 2.4× overall efficiency.

  • Robust on unanswerable questions and table/chart-heavy documents.

Abstract

Long-document understanding requires locating sparse and heterogeneous evidence across hundreds of pages, yet existing systems remain limited by static retrieval and fragile cross-round memory. Mainstream single-round methods commit to a fixed top- k page set at the outset and struggle to recover from early retrieval errors; recent iterative approaches allow multi-round evidence acquisition, but they do not investigate the propagation mechanism of cross-round states, making it difficult to track the dynamic changes in page relevance. To address these limitations, we propose DocMemo, a memory-guided framework that formulates long-document reasoning as dynamic evidence exploration. DocMemo maintains a tri-level retrieval state consisting of Document Schema Memory, Page Belief Memory, and Question Episodic Memory, which respectively capture structural priors, dynamic relevance estimation, and query-specific reasoning trajectories. During reasoning, DocMemo continuously refines cross-round page selection through Bayesian page belief updating with Thompson sampling, spatial proximity propagation, and structure-aware adaptive-granularity evidence access, while supplementing page-level evidence with fine-grained visual regions. Experiments on 3 benchmarks show that DocMemo achieves state-of-the-art performance and validate the efficacy of structured memory and dynamic page belief updating. Code is available at https://github.com/Harrygof/DocMemo.

Sources

Related papers