Reconstructing Implicit Scientific Knowledge: Evaluating LLM Agents through End-to-End Reproduction of Astronomy

arXiv:2609.35900 · astro-ph.IM, astro-ph.GA, cs.AI, cs.MA · Submitted 2026-09-28 · Read on arXiv

astro-ph.IM, astro-ph.GA, cs.AI, cs.MA

Submitted: 2026-09-28

Updated: 2026-09-28

Code: https://github.com/NousResearch/hermes-agent

Terminology

• Verification — 8× in this paper · explained in 2 episodes
• Foundation Model — 6× in this paper · explained in 3 episodes
• Benchmark — 4× in this paper · explained in 2 episodes
• Foundation Models — 3× in this paper · explained in 7 episodes
• Provenance — 3× in this paper · explained in 4 episodes
• Calibration — 2× in this paper · explained in 7 episodes
• Core — 2× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Multi-Agent Framework — 2× in this paper · explained in 3 episodes
• Agentic System — 1× in this paper · explained in 2 episodes
• Anisotropy — 1× in this paper · explained in 3 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Multi-Agent System — 1× in this paper · explained in 6 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Language Models are Few-Shot Learners
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• GPT-4 Technical Report
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Ultrafast switchable polar and magnetic orders by nonlinear light-matter interaction
• Path Following and Empirical Bayes Model Selection for Sparse Regression
• Attention Is All You Need
• AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
• ReAct: Synergizing Reasoning and Acting in Language Models

Related papers

• A signal dedispersion algorithm for imaging-based transient searches
• AVICA: A fully automated CASA pipeline for large volume VLBI data calibration
• Spectral Map Making with SPHEREx
• Long-Integration Magnetar Burst Observatory (LIMBO): Instrument Summary and Early FRB Rate Constraints
• Towards independent event horizon imaging of the supermassive black holes in M87 and the Milky Way
• A PINK update: Improvements to the CELEBI fast radio burst data reduction and analysis pipeline