Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

arXiv:2608.30047 · cs.AI · Submitted 2026-08-30 · Read on arXiv

cs.AI

Submitted: 2026-08-30

Updated: 2026-08-30

Code: https://github.com/shitanshubhushan/Creativity-Evaluation-Framework

Terminology

• LLM-as-a-Judge — 20× in this paper · explained in 3 episodes
• Core — 10× in this paper · explained in 2 episodes
• Benchmark — 7× in this paper · explained in 2 episodes
• Embeddings — 7× in this paper · explained in 2 episodes
• LLM judge — 5× in this paper · explained in 2 episodes
• Adam — 4× in this paper · explained in 2 episodes
• Trace — 4× in this paper · explained in 2 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Cosine Similarity — 3× in this paper · explained in 3 episodes
• Feature Extraction — 3× in this paper · explained in 2 episodes
• Reasoning Traces — 3× in this paper · explained in 3 episodes
• Regularization — 3× in this paper · explained in 3 episodes
• Transfer Learning — 3× in this paper · explained in 10 episodes
• AI Agents — 2× in this paper · explained in 2 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Accuracy — 2× in this paper · explained in 1 episode
• Commentary — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Iterative Refinement — 2× in this paper · explained in 3 episodes
• Scaffolding — 2× in this paper · explained in 3 episodes
• Action Space — 1× in this paper · explained in 1 episode
• Agentic Search — 1× in this paper · explained in 1 episode
• Annotator Agreement — 1× in this paper · explained in 1 episode
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• Feature Engineering — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Loss Function — 1× in this paper · explained in 2 episodes
• RAG — 1× in this paper · explained in 3 episodes
• Resource Constraints — 1× in this paper · explained in 3 episodes
• Semantic Similarity — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Vector Representations — 1× in this paper · explained in 2 episodes

Sources

• What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
• AGC-Bench: Measuring Artificial General Creativity
• Divergent Creativity in Humans and Large Language Models
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
• CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation
• MMTEB: Massive Multilingual Text Embedding Benchmark
• CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
• AIDE: AI-Driven Exploration in the Space of Code
• AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations
• EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery
• Kosmos: An AI Scientist for Autonomous Discovery
• AlphaEvolve: A coding agent for scientific and algorithmic discovery
• MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
• OpenAI GPT-5 System Card
• GLM-5: from Vibe Coding to Agentic Engineering
• Qwen3 Technical Report
• AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection