Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research

arXiv:2608.26753 · cs.SE, cs.AI · Submitted 2026-08-27 · Read on arXiv

cs.SE, cs.AI

Submitted: 2026-08-27

Updated: 2026-08-27

Code: https://github.com/Flavorfish/AutoRepro

Terminology

Hallucination — 44× in this paper · explained in 3 episodes
Verification — 42× in this paper · explained in 2 episodes
Benchmark — 39× in this paper · explained in 2 episodes
Alignment — 20× in this paper · explained in 1 episode
Ablation — 18× in this paper · explained in 1 episode
Core — 12× in this paper · explained in 2 episodes
Fidelity — 11× in this paper · explained in 2 episodes
Verifier — 11× in this paper · explained in 2 episodes
Accuracy — 9× in this paper · explained in 1 episode
Failure Modes — 7× in this paper · explained in 3 episodes
RAG — 7× in this paper · explained in 3 episodes
Ablation study — 5× in this paper · explained in 1 episode
Adaptation — 5× in this paper · explained in 1 episode
Aggregation — 5× in this paper · explained in 1 episode
Generalization — 4× in this paper · explained in 13 episodes
Large Language Model — 4× in this paper
Transformer — 4× in this paper · explained in 2 episodes
Agent Architectures — 3× in this paper · explained in 1 episode
Failure Taxonomy — 3× in this paper · explained in 2 episodes
Fine-Tuning — 3× in this paper · explained in 5 episodes
AI Agents — 2× in this paper · explained in 2 episodes
AUROC — 2× in this paper · explained in 2 episodes
Abstraction — 2× in this paper · explained in 1 episode
Alignment Verification — 2× in this paper · explained in 1 episode
Consistency — 2× in this paper · explained in 3 episodes
Embeddings — 2× in this paper · explained in 2 episodes
Linear probe — 2× in this paper · explained in 2 episodes
Multi-Agent System — 2× in this paper · explained in 6 episodes
Objective Function — 2× in this paper · explained in 2 episodes
Provenance — 2× in this paper · explained in 4 episodes
Resource Constraints — 2× in this paper · explained in 3 episodes
Robustness — 2× in this paper · explained in 5 episodes
Sparse Attention — 2× in this paper · explained in 5 episodes
Zero-Shot Transfer — 2× in this paper · explained in 4 episodes
Action Space — 1× in this paper · explained in 1 episode
Adjacency Matrix — 1× in this paper · explained in 1 episode
Agentic Science — 1× in this paper · explained in 1 episode
Agentic System — 1× in this paper · explained in 2 episodes
Attention Mechanism — 1× in this paper · explained in 3 episodes
Autonomous Research Catalyst — 1× in this paper
Contrastive Language-Image Pretraining — 1× in this paper
Contrastive Learning — 1× in this paper · explained in 5 episodes
Deep Q-Networks — 1× in this paper
Denoising Diffusion Implicit Models — 1× in this paper
Denoising Diffusion Probabilistic Models — 1× in this paper
Dense Passage Retriever — 1× in this paper
Graph Convolutional Networks — 1× in this paper
Hallucination Detection — 1× in this paper · explained in 2 episodes
Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
LoRA (Low-Rank Adaptation) — 1× in this paper · explained in 3 episodes
Loss Function — 1× in this paper · explained in 2 episodes
Multi-Agent Framework — 1× in this paper · explained in 3 episodes
Multi-agent Collaboration — 1× in this paper · explained in 2 episodes
Proximal Policy Optimization — 1× in this paper
RAG (Retrieval-Augmented Generation) — 1× in this paper · explained in 2 episodes
Region Proposal Network — 1× in this paper
Representation Learning — 1× in this paper · explained in 2 episodes
Retrieval-Augmented Generation — 1× in this paper
Scaffolding — 1× in this paper · explained in 3 episodes
Self-consistency — 1× in this paper · explained in 2 episodes
State Management — 1× in this paper · explained in 3 episodes
Structural Constraints — 1× in this paper · explained in 2 episodes
Trace — 1× in this paper · explained in 2 episodes

Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research

Terminology

Sources

Related papers