INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models
summary
The gist
The gist The benchmark reveals persistent gaps between parameter-level correctness and scientifically valid configurations, particularly in physically coupled settings such as DFT+U, magnetism, and
In short
INCARBench evaluates Large Language Models' ability to handle scientific configuration files for VASP calculations. It tests models on generating and repairing INCAR files across various materials and workflows, focusing on physical meaning (semantic), numerical choices (policy), and overall task success (TCC). The results show gaps between parameter accuracy and achieving scientifically valid configurations.
Key concepts
- Semantic Correctness
- This measures whether the intended physical meaning of a calculation is correctly encoded in the configuration file. It checks settings related to physical effects and workflow intent, such as choices for magnetism or electronic structure calculations. High semantic correctness means the model understands what physics it is trying to simulate.
- Policy Correctness
- This assesses whether the numerical and methodological choices within the INCAR file are appropriately configured. It evaluates parameters like energy cutoffs (ENCUT) or convergence criteria (EDIFF). This tests the model's ability to make sound computational decisions based on established scientific practices.
- Task-Critical Correctness (TCC)
- TCC is a strict metric where every requirement essential for a calculation to be valid is simultaneously met. A single error in any task-critical parameter group causes the entire case to fail. It represents the ultimate goal: producing a configuration that actually runs and yields the intended scientific result.
- Fix–Preserve Gap
- This metric compares how well a model corrects errors versus how well it preserves already correct settings during repair. A negative gap indicates that preservation (keeping good settings) is generally more successful than active error correction, suggesting models often prefer to maintain valid configurations over making modifications.
Terminology used across episodes
This episode discusses
- INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models · Paper Radio
- VASP Agent: An Agentic Framework for Autonomous First-principles Calculations
- VASPilot: MCP-Facilitated Multi-Agent Intelligence for Autonomous VASP Simulations
- OpenAI GPT-5 System Card
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Qwen3 Technical Report
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- Kimi K2.5: Visual Agentic Intelligence
- GLM-5: from Vibe Coding to Agentic Engineering
- MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science
- MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge
- LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction · Paper Radio
- MatTools: Benchmarking Large Language Models for Materials Science Tools
- DeepSeek-V3 Technical Report
- The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
The paper
INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models · Read on arXiv
Bin Shao, Jixiang Li, Xinyue Zhang, Baishun Yang, Zhiyang Liu, Weichao Wang
College of Electronic Information and Optical Engineering, Nankai University · School of Physics, Nankai University · Tianjin Key Laboratory of Optoelectronic Sensor and Sensing Network Technology, Nankai University · Shenzhen Research Institute of Nankai University
Large language models (LLMs) are increasingly being integrated into first-principles computational workflows, yet their ability to configure scientific calculations remains poorly understood. Here, we introduce INCARBench, a benchmark for evaluating LLMs on input configuration for the Vienna Ab initio Simulation Package (VASP) through both configuration generation and repair tasks. Evaluating 19 model configurations reveals substantial capability differences among current frontier models. While several models achieve high semantic and policy accuracy, task-critical correctness remains substantially lower, demonstrating that parameter-level correctness does not necessarily imply scientifically valid configurations. Failure analysis shows that errors concentrate in physically coupled settings involving DFT+ U, magnetism, and correlated materials, where multiple constraints must be satisfied simultaneously. Repair evaluation further reveals that correcting incorrect settings and preserving already-valid configurations are distinct capabilities, with configuration preservation remaining a major challenge. These findings establish scientific configuration as a measurable capability of large language models and provide a foundation for developing more reliable AI systems for computational materials science.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models".
Jane: Here's the gist of the paper. The gist The benchmark reveals persistent gaps between parameter-level correctness and scientifically valid configurations, particularly in physically coupled settings such as DFT+U, magnetism, and correlated materials.
Tom: Benchmark Design 1.
Jane: The generation task contains 48 materials drawn from 16 representative material families and covers a broad range of scientific scenarios commonly encountered in first-principles calculations, including metals, semiconductors, insulators, oxides, correlated materials, layered materials, magnetic materials, and battery materials The generation task contains 48 materials drawn from 16 material families and covers a broad range of scientific scenarios commonly encountered in first-principles calculations.
Title and authors: Tom: So, Jane, this paper is called INCARBench: A Benchmark for Scientific Configuration in VASP INCAR by Large Language Models. It’s about testing how well these models can actually set up the input files for a Vienna Ab initio Simulation Packag, which is VASP.
Jane: Yeah, so the core idea here is that we don't really know how good these models are at configuring scientific calculations yet because they have been used for all sorts of things before, but this paper says we need a proper way to test them on this specific task.
Lu: It addresses the fact that things like magnetism, DFT+U settings, and van der Waals interactions require different kinds of parameters to be set up correctly, and an executable INCAR file doesn't always mean the calculation is physically correct >
The paper's summary: Tom: The paper sets up this benchmark by taking forty-eight materials from sixteen different families, like metals or battery materials, and pairing each one with four different VASP workflow types <ref:2606.23571#pg3>. That gives us a total of one hundred ninety-two generation cases where the AI has to create an INCAR file <ref:2606.23571#pg3>.
Jane: And then they make it harder by creating repair tasks directly from those generation cases, resulting in five hundred seventy-six repair variants that include errors in the workflow or the physics itself > <ref:2606.23571#pg3>
Lu: The evaluation isn't just about whether you get the right answer; it’s broken down into three dimensions: semantic correctness, policy correctness, and task-critical correctness >
Tom: Semantic correctness checks if the AI understands what it means physically, like setting up a magnetic moment or a specific functional choice like LDAU >
Jane: Policy correctness is about the numerical and methodological choices, which are parameters like ENCUT or EDIFF that define how the calculation runs numerically >
Lu: And task-critical correctness is really strict; if any one of those task-critical parameters is wrong, the entire case fails because it invalidates what we were trying to calculate >
The paper's improvements: Tom: The authors of INCARBench suggest three main improvements to make this better. First, they want an accurate scientific configuration generation system that explicitly encodes physical intent into those semantic and task-critical requirements >
Jane: Then they propose a robust configuration repair agent that doesn't just fix errors but also distinguishes between error-fix rate and preservation rate to avoid making unnecessary changes to configurations that are already correct >
Lu: And finally, they suggest a failure mode localization module so the AI can diagnose where it’s failing by looking at which material families or challenge types are causing the most trouble, specifically pointing out areas like transition-metal oxides when DFT+U constraints are in play >
Conclusion: Tom: So, to wrap up on INCARBench, the main point is that parameter-level accuracy doesn't guarantee a scientifically valid configuration because of those gaps between semantic and task-critical correctness >
Jane: And it shows that models are often better at figuring out the right physical workflow even if they struggle with the exact numerical strategy for that workflow >
Lu: It establishes that scientific configuration is a capability of large language models and not just a formatting task, which means we need to think about these distinctions more clearly in how we build them >
Meng: From an engineering standpoint, seeing those five hundred seventy-six repair cases shows that preserving a valid configuration is often easier than fixing an injected error, which gives us a direction for building better repair agents > <ref:2606.23571#pg3>
Lalam: I'm pretty sure this whole INCARBench benchmark confirms what we suspect: the real difficulty lies in bridging that gap between understanding the physics and correctly applying the right numerical settings >
More episodes
- 2508.11850-LLM-Guided Evolution of Symbolic Acceleration Inequality Families for MILP
- 2508.20784-Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control
- 2508.20976-WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations
- 2510.02695-RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization
- 2510.03223-Attention-Aligned Reasoning for Large Language Models
- 2510.06919-Bayesian Nonparametric Dynamical Clustering of Time Series
- 2510.07073-VRPAgent: LLM-Driven Discovery of Heuristic Operators for Vehicle Routing Problems
- 2510.09717-Provable Training Data Identification for Large Language Models
- 2510.14697-Mitigating Model Merging Conflicts via Knowledge-Aware Task Vector Purification
- 2511.01553-Online Continual Learning on Intel Loihi 2 via a Co-designed Spiking Neural Network