DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation

DistilVDR: A Compact End-to-End Visual Document Retriever

via Dual-Student Distillation

Visual document retrieval (VDR) is dominated by multi-billion-parameter models that are slow to index at full corpus scale and expensive to serve. Prior compression routes either train a smaller multi-vector encoder from scratch or distil only the query side; neither yields a compact single-vector retriever end-to-end. We present DistilVDR, a 524 M end-to-end VDR system distilled bilaterally from a single 8 B vision-language teacher under a pointwise cosine alignment loss. All supervision comes from the frozen teacher’s embedding space, which was itself trained with relevance supervision, so the student objective needs no relevance labels, negative sampling, or contrastive term. We match VDR’s text-query and image-document input asymmetry with an asymmetric encoder-only student that concentrates visual capacity on the document side and keeps the query side at 70 M parameters. We release two variants that share the same encoders and training and differ only in the document encoder’s visual-tile budget: DistilVDR-HiRes attains 61.74 average NDCG@5 on ViDoRe v1+v2+v3 (86.9 % of the 8 B teacher) and leads every reproduced sub-1 B baseline on the high-resolution-sensitive v3 benchmark, while DistilVDR-Fast attains 59.98 at a 3× smaller visual-token budget. Both variants store one million documents in a 15.6 times smaller index than the strongest sub-1 B multi-vector baseline and index the corpus an order of magnitude faster. The code is available at https://github.com/Ryenhails/NanoVDR.

Improvements to AI systems:

1. Asymmetric encoder architecture for multimodal retrieval – Build AI systems where the query encoder is lightweight (e.g., 70M parameters) and the document encoder is heavier, concentrating visual capacity on the document side. This reduces inference latency and memory for user-facing queries while maintaining high retrieval quality for large document corpora.

2. Pointwise cosine alignment distillation without labels or negatives – Train compact student retrievers by distilling directly from a frozen teacher’s embedding space using a simple cosine loss. This eliminates the need for relevance labels, negative sampling, or contrastive learning, making training data-agnostic and dramatically simpler to scale.

3. Dual-student distillation with shared encoders – Generate multiple student variants (e.g., high-resolution vs. fast) from the same teacher by only changing the document encoder’s visual-tile budget, while keeping query and document encoders shared. This enables flexible deployment: a fast variant for real-time search and a high-res variant for accuracy-critical tasks, without retraining from scratch.

4. Single-vector end-to-end retrieval with compressed index – Replace multi-vector retrievers (which require large indexes) with a single-vector student that stores one million documents in a 15.6× smaller index and indexes an order of magnitude faster. This allows AI systems to handle corpus-scale retrieval on commodity hardware, enabling on-device or edge deployment.

5. Teacher-student transfer of relevance supervision – Leverage a large vision-language teacher (8B) that was trained with relevance supervision to implicitly transfer fine-grained relevance knowledge to a 524M student, without any explicit relevance labels at student training time. This enables building high-performing retrievers for new domains where labeled relevance data is scarce.

What the improved AI system can do:

• Real-time visual document search on mobile or edge devices with a 70M-parameter query encoder and a 524M document encoder, returning results in milliseconds for a million-document corpus.

• High-accuracy retrieval for high-resolution documents (e.g., scientific figures, medical scans, infographics) using the HiRes variant, achieving 86.9% of an 8B teacher’s performance while being 15× smaller and 10× faster to index.

• Deployable in low-resource environments – the 15.6× smaller index and order-of-magnitude faster indexing allow a single server to serve a large-scale document repository that previously required a cluster.

• Zero-label adaptation – given a new visual document collection (e.g., legal contracts, architectural drawings), the system can be distilled from any existing large VDR teacher without needing manual relevance annotations, enabling rapid domain transfer.

• Dynamic trade-off between speed and accuracy – by switching between DistilVDR-Fast and DistilVDR-HiRes at runtime (same encoders, different tile budgets), the system can adapt to varying query loads or latency requirements without retraining.

Visual document retrieval (VDR) is dominated by multi-billion-parameter models that are slow to index at full corpus scale and expensive to serve. Prior compression routes either train a smaller multi-vector encoder from scratch or distil only the query side; neither yields a compact single-vector retriever end-to-end. We present DistilVDR, a 524M end-to-end VDR system distilled bilaterally from a single 8B vision-language teacher under a pointwise cosine alignment loss. All supervision comes from the frozen teacher's embedding space, which was itself trained with relevance supervision, so the student objective needs no relevance labels, negative sampling, or contrastive term. We match VDR's text-query and image-document input asymmetry with an asymmetric encoder-only student that concentrates visual capacity on the document side and keeps the query side at 70M parameters. We release two variants that share the same encoders and training and differ only in the document encoder's visual-tile budget: DistilVDR-HiRes attains 61.74 average NDCG@5 on ViDoRe v1+v2+v3 (86.9% of the 8B teacher) and leads every reproduced sub-1B baseline on the high-resolution-sensitive v3 benchmark, while DistilVDR-Fast attains 59.98 at a 3 times smaller visual-token budget. Both variants store one million documents in a 15.6 times smaller index than the strongest sub-1B multi-vector baseline and index the corpus an order of magnitude faster. The code is available at https://github.com/Ryenhails/NanoVDR.