The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations

arXiv:2609.04047 · cs.IR, cs.CL · Submitted 2026-09-03 · Read on arXiv

cs.IR, cs.CL

Submitted: 2026-09-03

Updated: 2026-09-03

Comments: 30 pages, 2 figures, 19 tables. Substantially revised; supersedes the Research Square preprint 10.21203/rs.3.rs-8883056/v1. Includes a pre-registered external validation on three independent corpora (Motoki et al., Rozado, llm-stability)

Code: https://github.com/breckbaldwin/llm-stability

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Generalizability — 38× in this paper · explained in 6 episodes
• Cosine Similarity — 19× in this paper · explained in 3 episodes
• Consistency — 10× in this paper · explained in 3 episodes
• External Validation — 9× in this paper · explained in 2 episodes
• Calibration — 8× in this paper · explained in 7 episodes
• Accuracy — 7× in this paper · explained in 1 episode
• Benchmark — 7× in this paper · explained in 2 episodes
• Perplexity — 7× in this paper · explained in 3 episodes
• Core — 6× in this paper · explained in 2 episodes
• Embeddings — 6× in this paper · explained in 2 episodes
• Population Stability Index — 6× in this paper
• Non-stationarity — 4× in this paper · explained in 2 episodes
• Generative Model — 3× in this paper · explained in 2 episodes
• Aggregation — 2× in this paper · explained in 1 episode
• Large Language Model (LLM) — 2× in this paper · explained in 3 episodes
• Model Drift — 2× in this paper · explained in 4 episodes
• Prompt-Adjusted Share of Recommendation — 2× in this paper
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Anti-correlation — 1× in this paper · explained in 3 episodes
• Clustering — 1× in this paper · explained in 2 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Exchangeability — 1× in this paper · explained in 2 episodes
• Low-Resource Languages — 1× in this paper · explained in 4 episodes
• Non-Gaussianity — 1× in this paper · explained in 2 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Speculative Decoding — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• How Large Language Models Source Brand Reputation Across Languages and Markets
• Who Owns the AI Recommendation? A Multi-Industry Empirical Map of Brand Category Ownership Across Large Language Models
• The Effect of Sampling Temperature on Problem Solving in Large Language Models
• The Language Blind Spot: How Query Language and Brand Recognition Tier Shape AI-Constructed Brand Reputation Across Twelve European Languages
• LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows
• Non-Determinism of "Deterministic" LLM Settings

Related papers

• The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
• SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
• MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
• RRCM: Ranking-Driven Retrieval over Collaborative and Meta Memories for LLM Recommendation
• Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval
• UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG