Rating the Pitch, Not the Product: User Evaluations of LLMs Reflect Expectations More Than Performance

arXiv:2607.05113 · cs.CL, cs.CY, cs.HC · Submitted 2026-07-06 · Read on arXiv

cs.CL, cs.CY, cs.HC

Submitted: 2026-07-06

Updated: 2026-08-31

Comments: Accepted to EMNLP 2026 Main Conference

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

Benchmark — 14× in this paper · explained in 2 episodes
LLM judge — 11× in this paper · explained in 2 episodes
Robustness — 3× in this paper · explained in 5 episodes
Accuracy — 2× in this paper · explained in 1 episode
Adam — 1× in this paper · explained in 2 episodes
Adaptation — 1× in this paper · explained in 1 episode
Benchmarking — 1× in this paper · explained in 3 episodes
Calibration — 1× in this paper · explained in 7 episodes
Consistency — 1× in this paper · explained in 3 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Generalizability — 1× in this paper · explained in 6 episodes
Intraclass Correlation Coefficient — 1× in this paper
Personalization — 1× in this paper · explained in 2 episodes
Systematic Bias — 1× in this paper · explained in 3 episodes

Sources

Related papers