When More Documents Hurt RAG: Measuring and Reducing Vector Search Dilution with Domain-Scoped Retrieval

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "When More Documents Hurt RAG".

Jane: The gist The precision–faithfulness paradox arises when scaling retrieval-augmented generation to large, heterogeneous document collections,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper called "When More Documents Hurt RAG: Measuring and Reducing Vector Search Dilution with Domain-Scoped Retrieval". What it claims is that standard retrieval augmented generation gets worse when you use really big, mixed collections of documents.

Jane: That’s the core idea, Tom. It points out that dense similarity in vector search starts failing when the document collection gets too large and too varied, and top-k retrieval ends up bringing back chunks that look semantically similar but are actually wrong for the specific context you need to answer a question.

Lu: The paper formalizes this failure mode as vector search dilution, defining it by an equation: delta of q k star equals one minus P global of q over P scoped of q, which is the fraction of retrieval set m sub r m equals belonging to the target category k star when retrieval ranges over all documents C, and P scoped of q is that same fraction when you only search within categories C k star >

Meng: So, what's the practical problem? The abstract gives an example from a Wyoming Department of Transportation corpus where scaling from fifty-four to one thousand one hundred twenty-eight documents, which is eighty-eight thousand nine hundred seven chunks, cut accuracy from seventy-five percent down to below forty percent <ref:2606.11350#pg1,Wyoming Department of Transportation corpus where scaling from 54 to 1,128>.

Tom: Exactly. It shows that even with hybrid dense plus sparse retrieval methods in place, this dilution happens. They found similar patterns across six other Department of Transportation organizations too.

Jane: And to test this, they set up something called MASDR-RAG for their controlled experiment over two hundred expert validated queries and five different LLM backbones and two different index stacks <ref:2606.11350#pg1>.

Lalam: In that setup, domain scoping is the load bearing component of the system architecture >

Lu: They found that hard partitioning, which is one way to scope retrieval using metadata like Neo4j filters, isn't safe. But they found that training free soft scoping is safe when the structure of those scopes actually exists in real life >

Meng: That’s interesting because it means you don't need to manually curate all that metadata just to get better results; you can discover these relevant scopes without doing all the heavy lifting upfront >

Tom: But then they ran into something else, which they call a precision faithfulness paradox when using multi-agent orchestration, regardless of how you control the format or the prompt.

Jane: That paradox is what makes this paper so important because it tells us what to prioritize for enterprise scale RAG systems. They suggest that the main fix is simple: scope retrieval first, then just perform a single synthesis call whenever you can >

Lu: They show that in their evaluation, domain scoping using organizational metadata significantly improved the precision at ten from zero point seven seven up to zero point eight six across five different LLM backbones and six corpora >

Meng: But they also showed that when you use multi-agent orchestration under a Gemini production stack, the faithfulness score actually dropped from zero point six one down to zero point three five because the evidence retrieved contains duplicate passages or near duplicates >

Tom: So the main point here is that while using domain scoping helps with retrieval quality, orchestrating multiple agents to synthesize answers across those retrieved chunks creates a problem with accuracy and faithfulness >

Jane: It means for most organizations, especially those working with large, heterogeneous documents like government records or technical manuals, you should focus on getting that initial scope right before you start layering on complex multi-agent logic >

Lu: The paper generalizes these findings across different models and indexing implementations, showing the preference for domain scoping is robust regardless of the specific model or how you build your index >

Meng: From a production standpoint, this suggests that for single organization corpora with stable scopes, using something like R2-ROUTED with a single synthesis call gets the highest correctness and recall at half MASDR-RAG’s LLM call budget >

Tom: So the practical advice is to scope first, then stick to one synthesis call for now. It steers us away from complex multi-agent setups unless you have genuinely multi domain corpora that need it >

Jane: Exactly. The paper suggests that the load bearing decision for enterprise scale RAG should be domain scoping if you are dealing with a single organization's documents >

Lu: The generalization part is key because they tested this on a nine source public composite corpus and on HotpotQA distractors, showing the dilution effect isn't just a Wyoming Department of Transportation thing >

Meng: One limitation they state is that the routing taxonomy is manually crafted, and it assumes you have explicit organizational metadata already during the ingestion process >

Tom: And another limitation is that the open source evaluations are limited to models with seven to eight billion parameters, so they haven't tested how this holds up on much larger architectures yet >

Jane: So what we leave with is that domain scoping using organizational metadata seems like a load bearing decision for RAG systems dealing with large document collections, but you have to be careful about how you orchestrate the final synthesis step.

Conclusion: Tom: So we're wrapping up this look at "When More Documents Hurt RAG: Measuring and Reducing Vector Search Dilution with Domain-Scoped Retrieval".

Jane: Basically, it’s about how scaling up your document collection for a retrieval augmented generation system actually makes the search less accurate.

Lu: The authors are looking at how dense similarity in vector search loses its ability to discriminate when you have a lot of different documents mixed together.

Meng: They found that adding more documents, even in structured systems like those used by the Wyoming Department of Transportation, really hurts precision.

Lalam: It’s not just about finding similar stuff; it’s about finding the *right* kind of similar stuff for the specific context you need.

Tom: So what does this mean for a company building these systems? If you just throw more data in, your answer quality drops significantly.

Jane: The fix they propose is domain scoping. You restrict the search to only the documents relevant to that specific organizational area using metadata instead of searching everything at once.

Lu: They found that hard partitioning with metadata filters wasn't safe, but soft scoping, where you let the structure of those categories guide it without pre-curated labels, worked when you tested it against real-world structures.

Meng: That’s a relief for deployment because we don't have to manually tag every single document perfectly before we can use that scoping.

Lalam: And they showed that this domain scoping method significantly boosted precision at ten across five different large language models and six different test sets.

Tom: So, the main point is that before you start complex multi-agent stuff for synthesis, you need to lock down that initial retrieval step with a solid scope.

Jane: That’s right. The paper points to domain scoping as the load bearing decision for enterprise scale RAG systems when dealing with large document collections from a single source.

Lu: It suggests that for many real-world scenarios, focusing on getting that organizational boundary correct is more important than trying to get perfect synthesis across too many agents immediately.

Meng: So we should probably look at implementing some form of metadata filtering before we spend time optimizing the multi-agent orchestration layer.

Lalam: If you’re dealing with a single organization's documents, using a trained router like R2-ROUTED with just one synthesis call seems to give you the highest correctness and recall at half the cost of running all those agents.

Tom: It sounds like the future direction here is moving away from just more data toward smarter, scoped data access.

Jane: Exactly. And if you want to see how this applies beyond government DOTs, we’ll be looking at how these scope mechanisms generalize across different document types and organizational structures next.