Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations

The paper challenges the dominant retrieval-augmented generation (RAG) design — chunk the text, embed the chunks, and surface the top-k nearest neighbours of the query — arguing that for an important class of documents (financial statements, audit reports, regulatory returns) this design is structurally unsound, and the authors make the argument measurable.

The motivating question: asked of the Gujarat Finance Accounts 2024–25, Volume I, what is the difference between the Fiscal Deficit and the Revenue Surplus for 2024–25? The two operands sit 47 lines apart in the same statement, and their difference appears nowhere in the document — it must be computed. A tuned dense retriever failed to place both operands in the same top-k context on any trial, and the generator correctly declined to answer. The authors characterize this as retrieval declining to deliver, with the worse case being a retriever that surfaces one plausible figure and a fluent generator that uses it.

The testbed is a 780-page government financial report (Gujarat Finance Accounts 2024–25, Volume I, a public record by the Comptroller and Auditor General of India). Three measured properties carry the argument:

• It is a table, not prose. 86.8% of non-empty lines are table rows, with a median of 7 columns (p50/p90 columns: 7/10). The document has 19,198 non-empty lines, 1,683,730 characters, 189 statement/section headers, 16,664 table rows.

• Its numbers are near-duplicates. 58,791 numeric tokens collapse to 15,960 distinct values, a repetition ratio of 3.68. Thousands of similar magnitudes, in similar row contexts, compete in a single embedding space.

• Its units are inherited, not local. "The book declares units in headers of the form '(Rs. in lakh)': 666 lines declare lakh and 50 declare crore. A figure inherits the unit of the nearest declaration above it, at a distance of median 13 lines, p90 26 lines, max 143 lines. Lakh and crore differ by a factor of 100, so a retrieval unit that does not contain the governing header therefore yields figures that are ambiguous by two orders of magnitude — and, unlike a missing answer, this failure is invisible to the generator."

The paper defines a chunk as unitless if no line in it declares a unit, and yearless if no line matches a fiscal-year column header. Findings across a 5× range of chunk size (800 to 4,000 characters):

• Naive chunking degrades catastrophically at small sizes: at 800 characters, 67.4% of numeric chunks carry no unit declaration. Smaller chunks are usually recommended for precise retrieval; here they are precisely what destroys the context that makes a figure interpretable.

• The steelmanned fix works cheaply: a table-aware chunker that searches backwards up to 200 lines for governing unit and year rows and prepends them reduces unitless chunks from 18.0% to 0.3% at 2,000 characters — a factor of 60.

• The load-bearing observation: "the yearless residue does not respond to tuning. Across a five-fold range of chunk size it moves between 27.2% and 30.1%, a spread of 2.9 points, with no trend that tuning could exploit. Roughly 28% of numeric chunks cannot be attributed to a fiscal year no matter how the partition is drawn, because in this document the year is declared once for a table that runs across many pages. A partition fixed before the query is known has no way to recover it."

READ is an MCP server exposing three deterministic operations — normalized lexical search, structural navigation, and bounded span reads as four tools: memory list (files with sizes), memory outline (headings with line numbers), memory grep (normalized line search), and memory read (numbered lines, capped at 400 per call). There is no embedding model, no vector index, and no learned component; the server's entire state is the document text. The agent composes these in a closed loop — hypothesise a lexical anchor, search, inspect hits against the outline, read a span wide enough to capture the governing headers. Crucially, "READ is span-addressed, not chunk-addressed: the agent chooses [a, b] after seeing where its evidence lies, so it can extend upward to include a unit header. Interface A [top-k] cannot, because the partition was fixed offline. Every operation is a pure function of the document and arguments, so a trajectory is a replayable audit trail, not an opaque similarity score."

A literal grep over converted PDF text is materially weaker than over clean text. The converter introduces: "4,685 rows with a spurious leading empty column; 32,404 emphasis markers welded to tokens; 1,719 words split across table cells; 1,032 repeated page headers; and 24 distinct spellings of statement numbers across 654 mentions. Most consequentially, 14,269 numbers are printed with digit grouping. Probing twelve realistic queries, normalization changed the outcome for exactly two artifact types: digit grouping (e.g., 48964.89 matching printed 48,964.89) and words split across table cells. The first row is the important one. On this document the agent's queries are figures... and an agent that has derived or recalled a number writes it without separators."

The paper quantifies conversion damage: 483 total instances across five classes (suspect bare integer missing decimal: 174; orphaned separator cell: 139; word split across cells: 133; detached separator: 22; decimal detached from its digits: 15), affecting 440 table rows (2.6%) and 381 numeric rows (3.0%), with 11 numeric rows (0.09%) carrying an outright magnitude error. The clearest case: Statement No. 18, which lists loans sanctioned in perpetuity. The printed interest rates are 11.00 and 10.00 per cent. The conversion places the digits in one table cell and the decimal point in the cell below — an agent retrieving those lines still reads 1100 where the document prints 11.00. The authors argue this is a ceiling, not a confound. No retriever — dense, sparse, or agentic — can recover a decimal point that is in a different cell from its digits, so such questions get their own category (conversion limited) rather than being charged to whichever system was under test.

• Systems: All share one backbone (gemini-2.5-pro), one answer format, one accounting of tokens/cost/latency. Key entries: READlit (literal grep, isolating normalized matching), AgenticVec (the critical control: an agent with the same loop and budget but a vector-search tool instead of READ's operations — isolating interface from iteration), dense (bge-base-en-v1.5, table-aware chunks), BM25, hybrid (RRF + cross-encoder reranker), plus LongCtx (whole document in context) and Oracle baselines.

• Benchmark: 51 verified questions across six categories: single-figure lookup (24), cross-statement arithmetic (9), structural navigation (6), multi-row aggregation (5), unanswerable probes (5), conversion-limited items (2). Gold answers are transcribed from the document by the authors and then checked mechanically: every expected figure must appear within one line of its declared gold line, and column mappings are verified against an arithmetic identity the table itself satisfies. Arithmetic questions carry a validator asserting the derived value does not appear anywhere in the corpus — otherwise the question is a lookup in disguise.

• Metrics: Three independent signals: exact-figure accuracy (deterministic numeric comparison with 0.5% relative tolerance after normalizing Indian digit grouping and lakh/crore units), an LLM judge, and a mechanical groundedness check (Does every figure in the answer appear in text the system actually retrieved or read?) that a fluent-but-unsupported answer cannot pass. Comparisons are paired with exact McNemar tests, Holm-corrected across baselines.

Main comparison. READ answers 58.8% against dense retrieval's 15.7% — a paired difference of 43.1 points that survives correction at pHolm = 2 × 10−5. Against the hybrid pipeline: +29.4 points (p = 0.016). The most informative comparison is agentic RAG (same agent, same backbone, same turn budget, but a top-k tool): 27.5%, a 31.4-point gap (p = 0.012). Iteration alone therefore does not close the gap... The gain belongs to the interface, not to the loop.

Tuned dense baseline. The reported 15.7% is at k=8 (matched depth). Sweeping chunk size and depth finds dense's best at 35.3% (2,000-character chunks, k=16) — more than double the matched figure. "The mechanism is the one Section 4 predicts: a top-k system cannot ensure a chunk arrives with the header that gives its figures a unit and a year, so its only recourse is to retrieve more chunks and cover the header by chance. READ still beats that best configuration by 23.5 points (pHolm = 0.017), and every one of the six configurations we swept is beaten significantly."

What the evidence does not support. BM25 is not beaten. At 51.0% it sits 7.8 points below READ with a confidence interval spanning zero (p = 1.00). With 51 questions the minimum detectable difference is 27.7 points, so this is a null result, not evidence of equivalence. Of 22 discordant questions, 13 favour READ and 9 favour BM25 — close to a coin toss. The honest reading is that our evidence separates embedding-based retrieval from embedding-free retrieval, and does not separate the two embedding-free systems from each other. BM25 also costs a third as much as READ and runs in a third the latency.

Ablations. Both ablations score above READ: removing outline or replacing normalized matching with raw regex each yields 66.7% against READ's 58.8%, but neither difference approaches significance (p = 1.00). We therefore cannot claim the full operation set is the best configuration — only that it is not detectably worse.

Groundedness. "READ is not the most grounded system. At 58.0% it trails BM25 (67.3%) and hybrid (72.0%). A system that reads more text has more opportunity to state a figure that its own retrieved text does not contain... This cuts against the interpretability argument and we report it as such."

Where the difference comes from. The gap is concentrated in lookup: 70.8% for READ against 12.5% for dense — precisely where thousands of near-identical numbers compete in one embedding space. On navigation every system does comparatively well, because the answer is carried by prose and headings rather than by digits. The failure taxonomy: Dense retrieval fails to place the gold evidence in context on 13 questions and hybrid on 15, against READ's 4. Dense declines to answer 11 answerable questions, hybrid 12, READ 1. Across the whole benchmark, dense retrieval surfaces a gold page in only 4.3% of questions, against READ's 41.3%. Agentic RAG shows the opposite profile: it rarely fails to surface something (5 never-surfaced) but produces 21 ungrounded answers, the worst of any system.

Unanswerable probes. Five questions have no answer in the document. Dense retrieval answers four of them anyway (80%), hybrid three, READ and BM25 two. The two systems that never fabricated were naive-grep and agentic RAG at 0% — but for agentic RAG this is not a virtue — it also abstains on 8 answerable questions, so it is simply a reticent system.

Cost. "READ is the most expensive system per question (0.058 against dense's 0.023) and the slowest (31.4 s against 12.7 s), because it makes 4.8 tool calls per question and reads substantially more text... The case for READ rests on the failure taxonomy and on auditability, not on the cost–accuracy frontier."

An n=2 pilot on the motivating question showed the dense baseline abstained on every cell, while READ located both and computed the difference, Rs. 30,021.95 crore, a value that appears nowhere in the document. READ consumed 58–78k input tokens against dense's 6.7–8.2k — roughly an order of magnitude. The paper also records a methodological caution: an apparent provenance failure (wrong statement citation) turned out to be the authors' own parsing bug, not the model's error. The lesson is not about the model. It is that a parsing bug in the analysis layer can manufacture a confident, plausible finding about model behaviour.

The authors are explicit: One document, one backbone, 51 questions... Nothing establishes that the result transfers to another document class. The benchmark was built by the authors of the system under test and grew while the experiment ran, and the growth was not blind — an earlier 25-question version produced no significant differences, and two batches were added after inspection revealed coverage defects. They report dense accuracy at each stage (24.0%, 16.7%, 15.7%) so the drift is visible rather than hidden, and concede A reader should treat the n=51 result as one we sought and found, not one we stumbled into. The conversion ceiling is a property of pymupdf4llm on this PDF, not a universal constant.

The narrow claim: for documents whose meaning depends on inherited layout context, a partition fixed before the query is the wrong primitive. The paper's central finding is that on this document, the vector index is not merely unnecessary — it is the component doing the damage, and a practitioner with a lexical index and no agent would already have captured most of the available gain. The framing conclusion: "As models become competent agents, the retrieval question shifts from which retriever ranks best to which interface lets the agent see, verify, and account for the evidence. For long structured documents we argue the answer is the simplest one: let the model read."

Here are the specific improvements I would make to an AI system based on this paper, followed by what the resulting system can do.

1. Make document structure a first-class routing signal — before any retrieval happens.

Measure the document before choosing an architecture: table-line ratio (this paper: 86.8%), numeric near-duplicate ratio (3.68× repetition), and header-inheritance distance (median 13 lines, p90 26, max 143). If the document is table-heavy with near-duplicate figures and inherited units, route it away from embedding-based top-k entirely and toward deterministic, span-addressed tools. This is the single highest-leverage change: the paper's central finding is that the vector index is the component doing the damage, not a missing piece.

2. Give the model span-addressed reading tools instead of pre-cut chunks.

Replace retrieve k chunks with three deterministic operations: outline (headings with line numbers), grep (normalized lexical search), and read [a, b] (bounded span, capped at 400 lines). The agent chooses [a, b] after seeing where its evidence lies, so it can extend upward to capture a governing unit header that sits up to 143 lines above the figure. A top-k interface cannot do this because the partition was fixed offline. The paper's critical control proves this matters: the same agent, same loop, same budget, but a vector-search tool instead of READ's tools, scored 27.5% vs 58.8% — the gain belongs to the interface, not the loop.

3. Build a PDF-normalization layer spec'd to the corruptions that actually matter, not generic text cleaning.

The paper measured which conversion artifacts change retrieval outcomes: digit grouping (14,269 numbers printed with separators like 48,964.89 vs the agent's 48964.89) and words split across table cells. Also normalize the other measured noise: 32,404 emphasis markers welded to tokens, 4,685 spurious leading empty columns, 1,032 repeated page headers, and 24 distinct spellings of statement numbers across 654 mentions. This is cheap and turned naive grep into a system within statistical parity of READ.

4. Sew header inheritance into the document at ingest time.

Walk each table row backward (up to 200 lines) to find the governing unit declaration and the fiscal-year column header, and attach them to the row. The paper shows this takes unitless chunks from 18.0% to 0.3% at 2,000 characters — a 60× reduction — and it works even if you keep a dense pipeline. But note the hard floor: 28% of numeric chunks are yearless at every chunk size because the year is declared once for a multi-page table. So header-sewing fixes units; year attribution must be handled dynamically by span reads, not by any fixed partition.

5. Add a mechanical groundedness gate that fluency cannot pass.

After drafting an answer, deterministically check that every figure in the answer appears in the exact text the system read. If any figure is ungrounded, force a revision or abstention. This is a hard, auditable constraint rather than an LLM-judge heuristic. The paper shows why it's necessary: Agentic RAG (top-k tool + iteration) produced 21 ungrounded answers, the worst of any system, precisely because a fluent generator will confidently use a plausible near-duplicate figure. The gate also fixes hallucination on unanswerable probes: dense retrieval fabricated answers to 4 of 5 questions with no answer in the document; the improved system should abstain. But calibrate abstention — agentic RAG fabricated 0% but also abstained on 8 answerable questions. The gate should be triggered only by failed groundedness, not by default reticence.

6. Detect conversion damage and route it to its own category instead of answering.

Add a pattern detector for the five measured damage classes: suspect bare integers missing decimals (174), orphaned separator cells (139), words split across cells (133), detached separators (22), and decimals detached from digits (15). The clearest case: the document prints 11.00%, the conversion places digits in one cell and the decimal in the cell below, and any retriever reads 1100. Since no system can recover a decimal point that lives in a different cell, the correct behavior is to flag the item as conversion-limited and tell the user, not to answer confidently. This is a ceiling, not a confound — handle it explicitly.

7. Use a cost-laddered fallback with BM25-level search as the default starting point.

BM25 scored 51.0% vs READ's 58.8% — a null result, not a real gap — while costing a third and running in a third the latency. So the improved system should begin with normalized lexical search, which is deterministic, replayable, and cheap, and escalate to wider span reads only when the cheap pass fails the groundedness gate. Don't pay 4.8 tool calls and 0.058/question unless the evidence requires it.

8. Log every operation as a replayable audit trail.

Because grep, outline, and read are pure functions of (document, arguments), every trajectory is a reproducible citation chain: search query → hit line numbers → spans read → groundedness verdict. This converts retrieval from opaque similarity score to verifiable evidence trail, which is the property that matters for regulatory, audit, and financial documents.

• Answer cross-statement arithmetic that appears nowhere in the document. Given what is the difference between Fiscal Deficit and Revenue Surplus?, it locates both operands via normalized lexical anchors, extends its span upward to capture the governing unit and year headers, reads enough context across the multi-page table, computes the result (Rs.