Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation

arXiv:2610.02022 · cs.CL · Submitted 2026-10-01 · Read on arXiv

cs.CL

Submitted: 2026-10-01

Updated: 2026-10-01

Code: https://github.com/allenai/asta-paper-finder

Terminology

• Aggregation — 34× in this paper · explained in 1 episode
• Benchmark — 28× in this paper · explained in 2 episodes
• Accuracy — 24× in this paper · explained in 1 episode
• Clustering — 6× in this paper · explained in 2 episodes
• Ablation — 5× in this paper · explained in 1 episode
• Consistency — 5× in this paper · explained in 3 episodes
• RAG — 5× in this paper · explained in 3 episodes
• Core — 4× in this paper · explained in 2 episodes
• Evaluation Dimensions — 4× in this paper · explained in 3 episodes
• Hallucination — 4× in this paper · explained in 3 episodes
• LLM judge — 4× in this paper · explained in 2 episodes
• Generative Model — 3× in this paper · explained in 2 episodes
• Generative Models — 3× in this paper · explained in 2 episodes
• Regularization — 3× in this paper · explained in 3 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Decoupling — 2× in this paper · explained in 2 episodes
• Feature Extraction — 2× in this paper · explained in 2 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Multi-Agent System — 2× in this paper · explained in 6 episodes
• Pseudo-labeling — 2× in this paper · explained in 3 episodes
• Representation Learning — 2× in this paper · explained in 2 episodes
• Wasserstein Distance — 2× in this paper · explained in 4 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Adversarial Attacks — 1× in this paper · explained in 1 episode
• Attention Mechanism — 1× in this paper · explained in 2 episodes
• Commentary — 1× in this paper · explained in 2 episodes
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• Diffusion Model — 1× in this paper · explained in 8 episodes
• Diffusion Models — 1× in this paper · explained in 6 episodes
• Diffusion Process — 1× in this paper · explained in 2 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Embodied AI — 1× in this paper · explained in 2 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Meta-learning — 1× in this paper · explained in 2 episodes
• Model-Agnostic — 1× in this paper · explained in 2 episodes
• Multi-agent system (MAS) — 1× in this paper · explained in 2 episodes
• Pairwise Comparisons — 1× in this paper · explained in 2 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
• Measuring the Gap Between Human and LLM Research Ideas
• Accelerating scientific discovery with Co-Scientist
• All That Glitters is Not Novel: Plagiarism in AI Generated Research
• Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas
• Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents
• Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications
• An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics
• ScholarEval: Research Idea Evaluation Grounded in Literature
• InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
• Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation
• Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
• Is this Idea Novel? An Automated Benchmark for Judgment of Research Ideas
• Literature-Grounded Novelty Assessment of Scientific Ideas
• Unlocking LLM Creativity in Science through Analogical Reasoning
• The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
• Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
• Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System
• The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
• Large Language Models are not Fair Evaluators

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering