2608.07067-DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding".

Jane: The paper was written by Hanshu Yao, Jianfeng Zhong, Niu Lian and Jinpeng Wang from Harbin Institute of Technology, Shenzhen and Tsinghua Shenzhen International Graduate School, Tsinghua University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We just spent the morning on a paper that made me want to reorganize my own desk. It tackles a nasty problem: answering questions from documents that run hundreds of pages, where the answer might hide in a table on page 87. Most systems grab a fixed pile of pages up front and then pray. This one keeps hunting.

Jane: And that hunting is the whole trick. The authors — Hanshu Yao, Jianfeng Zhong, Niu Lian, and Jinpeng Wang — call it memory-guided evidence discovery. Instead of one shot, the system goes round after round. It remembers which pages looked useful, which did not, and where the question has already taken it. Like a detective with a notepad.

Lu: The notepad has three layers. Document Schema Memory stores the stable structure: the document type and the section map. Page Belief Memory tracks how likely each page is relevant, and it shifts with feedback. Question Episodic Memory holds the clue trail for the current query. Stable knowledge, dynamic relevance, and local experience — the paper separates all three.

Tom: And they put real math underneath?

Lu: Real math. Every page carries a Beta distribution. Reasoning feedback updates those distributions through Bayesian belief updating. Thompson sampling then decides which pages deserve another look, balancing safe bets with uncertain leads.

Meng: It pays off on the leaderboards. MMLongBench-Doc lands at 71.3 percent, LongDocURL at 81.1, PaperTab at 80.4. Against the strongest agentic baselines, that is a 15-point jump on PaperTab.

Jane: Fifteen points on the table-heavy benchmark, which makes sense. The system zooms into high-resolution crops of tables and charts when the page-level view gets too blurry. They call it adaptive-granularity evidence access.

Lalam: Stepping back, we keep shoving documents into context windows and calling that reading. This paper treats reading as search with memory and feedback. It also does it cheaper — about 0.41 times the retrieval cost of a strong baseline for higher accuracy. Efficiency and accuracy moving together is rare, and that is why this matters beyond the benchmark tables.

Meng: The gains are not just averages either. Unanswerable cases jump to 78.8 percent, which is where most systems collapse into hallucination.

Tom: The code is already public, too, so this is not vaporware. Page one sets up the problem with a sharp critique of everyone else's approach.

Page 1 of the paper: Jane: We just sketched the headline results. Now page one explains why existing systems fail, and the opening argument lands hard. The central challenge is dynamic: locating, updating, and integrating relevant pages under a limited evidence budget. Passively reading a long input does not cut it.

Lu: Context budgets are the wall. A hundred-page PDF will never fit comfortably in a model window, so you have to choose pages. The paper's critique is that most systems choose once, then live with the consequences. Single-turn retrieval fixes a candidate set at the start, and if a key chart is missed in that first pass, you are done.

Meng: Then there are iterative methods that add more rounds. The paper credits them for trying but says they drop the ball on memory. Cross-round information is mostly preserved by rebuilding the context each time. That is closer to repeated independent retrieval than to actual evidence exploration.

Tom: That is the gap this system is built for. Retrieval that can recover from early mistakes. The authors frame the whole thing as dynamic evidence exploration, not passive reading.

Lalam: And the framing borrows from cognitive science. Complementary learning systems — the brain separates slow, stable knowledge in the neocortex from fast, flexible traces in the hippocampus. The paper takes that separation and builds it straight into the architecture.

Jane: One more thing from the intro: the evidence itself is messy. The answer can live in text, tables, figures, or visually structured layouts. A retriever that only searches keywords will starve.

Meng: Their phrase stuck with me — "stateful exploration." The model should carry a state of where it has been and what it half-learned, instead of starting from zero every round.

Tom: So the intro lines up three failure modes against the three memory layers. Static retrieval is rigid. Iterative retrieval forgets. The bet here is that structured memory fixes both. Then Figure 1 draws that contrast.

Jane: And that figure is not decorative — it is the entire thesis in one image. We should unpack it.

Page 2 of the paper: Lu: We left off at that thesis-in-one-image. Page two opens with the figure, and it draws three retrieval paradigms side by side. Single-turn retrieval is a fixed candidate pool feeding a one-shot pipeline — rigid, brittle. Iterative retrieval stacks context flat, like piling papers on a desk without sorting them. The proposed framework separates memory from evidence discovery and updates both continuously.

Tom: The figure even labels the memory layers with brain names. Prefrontal cortex for slow semantic memory, hippocampus for fast episodic memory. It is a metaphor, sure, but it guides the engineering.

Jane: And the captions tell a story on their own. "Rigid and One-Shot Pipeline" for the old way, "Flat Context Accumulation" for the iterative way. The labels alone show where the field went wrong.

Meng: Then come the three contributions. A tri-level memory framework. Bayesian page belief updating with Thompson sampling and spatial contiguity propagation. Experiments on three benchmarks showing consistent gains. Short list, heavy claims.

Lu: Related work starts right after. Static retrieval-augmented approaches pick evidence first and reason later. Some add visual retrievers, structured retrieval, or fine-grained localization, but the selection happens before reasoning begins. The paper's verdict: once key evidence is missed, later reasoning has little chance to recover it.

Jane: The second bucket is iterative retrieval, and the paper singles out SimpleDoc. SimpleDoc lets an agent keep retrieving when evidence is insufficient. But it lacks a mechanism for transferring states across rounds. The paper says that process is closer to repeating the same search than building on it.

Tom: So the literature review sets up a clean dichotomy. Static methods cannot adapt. Iterative methods do not remember. The method section starts on page three, with the memory definitions.

Lalam: What strikes me is the vocabulary. Document Schema, Page Belief, Question Episodic — that maps onto long-term structure, working state, and short-term experience. It gives researchers a language for what previous systems kept implicit.

Jane: And the phrase in the figure — "decouple memory and dynamic evidence discovery" — that might be the one-line summary of the entire paper.

Tom: It is. Page three then shows the schema layer getting built before any question arrives.

Page 3 of the paper: Meng: The figure gave us the map, and now page three fills in the details. It finishes the related work with agent memory, and that subsection is basically a complaint. LLM agents have explored hierarchical storage, long-term memory updating, associative organization. But most of that targets open-ended interaction or video-style experiences unfolding over time. It does not handle documents where text, figures, tables, and layout cues are deeply intertwined.

Jane: Exactly. A PDF is not a video. The signals are heterogeneous and they sit on the same page, physically entangled. The proposed tri-level memory is positioned as the missing piece for that mess.

Lu: Then the method section formalizes the setup. Document D with N pages, query q, up to T retrieval-reasoning rounds. Offline, page embeddings and summaries get computed once. Online, the memory state is written as a triple: schema, belief, episodic.

Tom: The first piece is Document Schema Memory, built offline and fixed during inference. Every page gets a summary, and the system aggregates them into one package: the document type, a structural index of page ranges with topic labels, and a global summary. That package is the navigation map.

Meng: So the schema is a map of the building, drawn before anyone asks a question. It says a survey report runs pages one through ten on demographics, then twenty to thirty on opinions. That map never changes, no matter who is asking.

Jane: And because it is query-independent, the cost gets amortized. Build it once, reuse it across every question on that document. That is a smart place to spend offline compute.

Tom: It also gives the retrieval agent a navigation prior. When a later query mentions unemployment, the schema can point toward the economy section before the visual retriever even runs.

Lalam: Persistent structure on one track, dynamic beliefs on another. That separation keeps the offline investment useful and the online state lean. The second memory layer is where the probability enters, and that is page four.

Page 4 of the paper: Lu: Page four is where the math arrives. Page Belief Memory gives every page a Beta distribution — two numbers that encode how strongly evidence supports the page being relevant versus irrelevant. The posterior mean becomes the page's accumulated relevance confidence.

Tom: So each page is basically a probability coin being flipped during retrieval?

Lu: Not flipped — sampled. The system draws from that distribution when choosing pages. That is the Thompson sampling bit, and it is coming on page five.

Meng: The initial beliefs come from the visual retriever, ColQwen2.5. It computes token-level similarity between the query and each page using a late-interaction mechanism, normalizes the score, and converts it into a Beta prior. A strength parameter S controls how much the visual signal is trusted — they set it to five.

Jane: But the elegant part is the update rule. After each reasoning round, the model reports useful pages and irrelevant pages. Useful feedback increments the alpha parameter; irrelevant feedback increments beta. Beta-Bernoulli conjugacy keeps the update a simple addition.

Tom: Then the spatial propagation step, and I want to underline this. The paper notices that evidence in long documents clusters locally. If page 20 is valuable, pages 18 through 22 deserve another look. So positive feedback spreads to neighbors within a small radius, with a decay factor.

Jane: Negative feedback does not spread. That asymmetry stops one bad page from poisoning the whole neighborhood. With a radius of two and a decay of 0.5, close pages get a real boost while far ones fade out.

Lu: The propagation turns document layout into a soft prior. Pages sit next to each other in physical space, and the model exploits that ordering.

Meng: And the paper grounds it in cognitive load theory — the spatial contiguity principle. Related information belongs close together, so the retrieval rule borrows from how humans learn.

Jane: By the end of page four, beliefs have been born from visual scores, updated by reasoning feedback, and spread across neighborhoods. Page five shows how those beliefs steer the next retrieval round.

Page 5 of the paper: Meng: Page five explains the retrieval loop. For each page, the system samples a relevance estimate from its Beta distribution. That sample is Thompson sampling, a classic bandit strategy that balances exploiting confident choices with exploring uncertain ones.

Jane: The sampled value blends with a fresh visual similarity score against the refined query. The blend weight lambda grows over rounds — the schedule is zero, 0.3, 0.6, 0.6. Early on, pure vision; later, accumulated belief takes the wheel.

Tom: After scoring, a candidate pool is built, and a language model reranks it using page summaries plus the schema and episodic memories. The refined set merges into an accumulated evidence set. High-confidence historical pages stay in the conversation.

Lu: Then the reasoner reads the evidence and makes a three-way decision. It answers when evidence is sufficient. It returns not_answerable when the document lacks support. Otherwise, it writes a refined query and a note, both stored into Question Episodic Memory. The loop repeats until an answer, a refusal, or the round limit.

Meng: That explicit unanswerable path deserves a pause. The system can say "this document does not contain it." The experiments show strong gains on those questions, where static retrievers tend to hallucinate instead.

Jane: The last mechanism on page five is adaptive-granularity evidence access. When a page is dense, like a table-heavy page, the system appends high-resolution crops to the full-page image. Full-page rendering loses the fine details; the crops bring them back.

Tom: And the reasoner is warned not to request the same pages or elements again. The episodic memory records what has been ruled out, so later rounds keep narrowing the search instead of looping.

Lalam: What stands out to me is that uncertainty is treated as a resource. Sampling gives the system a budget for exploring pages it is not sure about. That is the difference between ranking and deciding.

Jane: One more nuance — lambda starts at zero, so the first round is pure visual retrieval. The beliefs only earn their weight after the reasoner has something to say.

Tom: Conservative design, and smart. Trust the vision first, trust the memory after it has proven itself. Page six switches to benchmarks and experimental setup.

Page 6 of the paper: Lu: Page six lays out the experiments. MMLongBench-Doc has 1,082 questions across 135 documents, averaging 47.5 pages, with some reaching 112. Those questions deliberately mix text, images, tables, charts, layout understanding, and unanswerable cases.

Jane: LongDocURL is bigger — 2,325 question-answer pairs across 396 PDFs, demanding long-document understanding, numerical reasoning, and cross-element grounding. PaperTab offers 393 questions over 307 scientific papers, focused on tables. Three benchmarks, three distinct flavors of pain.

Tom: Evaluation uses GPT-4.1 as an automatic judge, scoring each prediction as correct or incorrect. But two extra metrics matter: Evidence Recall, the share of ground-truth evidence pages the system found, and All-Hit Rate, the fraction of questions where every annotated page was retrieved. That separates retrieval quality from answer luck.

Meng: The backbone is Qwen3.5-VL-9B, an open multimodal model, served with vLLM. ColQwen2.5 encodes pages offline, and MinerU extracts the table and figure crops. The hyperparameters fall into place: prior strength five, the lambda schedule we just mentioned, propagation radius two, decay factor 0.5.

Lu: Table 1 gives the MMLongBench breakdown. The system hits 71.3 overall, with 73.3 on tables and 78.8 on unanswerable questions. The table gain tracks the adaptive-granularity crops. The unanswerable gain tracks the accumulated memory state.

Jane: The competition is serious — GPT-4o, Claude-4-Sonnet, Gemini models, open MLLMs like InternVL3, agentic systems like SimpleDoc and DocLens. The proposed framework tops the entire table.

Lalam: One detail I appreciate: the evaluation protocol matches recent baselines, so the comparison is fair. The paper also validates the judge against human raters later, with 96.7 percent agreement. That diligence makes the numbers credible.

Tom: And the benchmarks were chosen to cover different evidence types — dense tables, long PDFs, scientific papers. That breadth is what makes the average score meaningful.

Meng: Also note the unanswerable category. Many systems collapse there because they refuse to say "not answerable."

Jane: And the paper treats that refusal as a first-class output, not a failure. Page seven asks the harder question: does it still win when everyone gets the same backbone and budget?

Page 7 of the paper: Jane: Page seven starts with a fairness check. They lock the backbone, retriever, page budget, and round count, then compare against SimpleDoc and MoLoRAG. The proposed system still wins — 61.7 over SimpleDoc's 60.1 with the smaller backbone, and 71.3 over 69.3 with the larger one.

Tom: The cross-benchmark table follows. The average across MMLongBench, LongDocURL, and PaperTab lands at 77.6. The strongest agentic baseline sits at 66.1. That is more than an eleven-point gap.

Lu: The ablations reveal where the credit goes. Removing Page Belief Memory drops accuracy from 71.28 to 68.80. Removing all memory modules drops it to 68.47. Removing Bayesian updating entirely lands at 68.80 again. The dynamic belief machinery is doing heavy lifting.

Meng: The retrieval metrics tell the same story. Evidence recall climbs from 28.32 percent in round one to 69.56 percent by round three. All-hit rate jumps from 12.9 to 58.05. The second iteration delivers the biggest leap, which shows the reasoning feedback from round one actually redirects the search.

Jane: Efficiency is the surprise punchline. SimpleDoc always runs three iterations; this system averages 1.24. That is 0.41 times the retrieval cost for higher accuracy. The paper packages it as a 2.4 times efficiency gain.

Tom: And they verified the judge too — 96.7 percent agreement with human raters, Cohen's kappa 0.92. The accuracy numbers are not an artifact of the evaluator.

Lu: The appendix also tests Thompson sampling head-to-head. Replace it with greedy selection and accuracy falls from 71.28 to 68.62. Uncertainty-based exploration carries the whole system.

Meng: Notice the ablation pattern. Removing one module costs a few points, removing all of them costs more. The pieces compound instead of overlapping.

Jane: So memory helps, exploration helps, and the gains survive controlled comparisons. Page eight then shows the whole system on a single concrete question, and that example is a lot of fun.

Page 8 of the paper: Meng: Page eight gives us a worked example, and it reads like a detective story. The query asks which country's youth show the greatest concern about unemployment. The document is the Arab Youth Survey 2014, 45 pages. The answer hides in a chart.

Jane: Round one retrieves pages 16 through 20. The reasoner reads them and concludes they hold general survey statistics but no per-country breakdown. The chart must be somewhere else. So it writes a refined query mentioning a stacked bar chart and updates its episodic note.

Tom: Round two widens the net — pages 43, 17,

Conclusion: Tom: So the big idea from this paper is that long-document reading works better when you treat it as a hunt with a memory, not a one-time glance.

Jane: Exactly. Three kinds of memory — the document's map, the page-by-page confidence, and the trail of clues you've already found.

Tom: And every round of reasoning writes back into that memory. The system gets smarter about where to look next.

Jane: That's what separates it from the old iterative methods. They just kept pulling pages; DocMemo actually learns from what it saw.

Tom: Those numbers on tables and unanswerable questions really sold me. Seventy-three and a half on tables, almost seventy-nine on unanswerable.

Jane: And it does it with fewer retrieval rounds than the baseline. More accuracy, less work. That's the rare combo.

Tom: The Bayesian belief updating felt like the hidden engine. Every page carries a little probability coin, and Thompson sampling flips it to decide where to explore.

Jane: Smart design. You don't just grab the top pages, you give the uncertain ones a chance to prove themselves.

Tom: The worked example with the Arab Youth Survey was a nice way to close. You could practically watch the system zero in on page twenty.

Jane: It looked like a detective correcting their own assumptions. First guess wrong, adjust, find the chart, done.

Tom: And the code is public, so anyone can try it. That lowers the barrier for the next round of work.

Jane: I hope someone pushes this toward even longer documents — maybe book-length reasoning, or multi-document comparisons.

Tom: Or applies the same memory structure to video understanding. That's a natural next step.

Jane: Good thought. But our time on this one is up.

Tom: We'll be back with something fresh. Until then, keep searching with a notepad.

Jane: See you next episode.