Offline A/B Testing of Slate Recommendation Systems with LLMs: Reducing the Dependency on Pre-Collected User Interaction Data

arXiv:2511.04541 · cs.IR, cs.AI · Submitted 2025-11-06 · Read on arXiv

cs.IR, cs.AI

Submitted: 2025-11-06

Updated: 2026-09-23

Terminology

• Regret — 21× in this paper · explained in 3 episodes
• World Model — 13× in this paper · explained in 7 episodes
• LLM-as-a-Judge — 7× in this paper · explained in 3 episodes
• Large Language Models — 7× in this paper
• Consistency — 6× in this paper · explained in 3 episodes
• Alignment — 5× in this paper · explained in 1 episode
• Axioms — 5× in this paper · explained in 2 episodes
• Reinforcement Learning — 4× in this paper · explained in 4 episodes
• Benchmark — 2× in this paper · explained in 2 episodes
• Pairwise Comparisons — 2× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• LLM judge — 1× in this paper · explained in 2 episodes

Related papers

• The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
• SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
• MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
• RRCM: Ranking-Driven Retrieval over Collaborative and Meta Memories for LLM Recommendation
• Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval
• UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG