SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

SimulCost: A Cost-Aware Benchmark for Automating Physics Simulations with LLMs

Abstract

"a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench."

"Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce S IMUL C OST, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. S IMUL C OST compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,643 single-round (initial guess) and 2,304 multi-round (adjustment by trial-and-error) tasks across 11 simulators from fluid dynamics, solid mechanics, and plasma physics, whose costs are analytically defined and platform-independent. A twelfth simulator, a production plasma code measurable only by wall clock, is reported separately. Frontier LLMs achieve 45–62% success rates in single-round mode, dropping to 34–50% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 66–81%, but LLMs are 1.5–2.7× slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source S IMUL C OST as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments."

Introduction

"Large Language Models (LLMs) show promise for scientific workflows through code generation, complex reasoning, and especially tool calling... By offloading computations to domain-specific tools, LLM agents reduce hallucinations through grounded outputs. This capability has accelerated workflows in scientific domains including chemistry, computational fluid dynamics, and data science."

"However, existing evaluations focus on task correctness and token costs while overlooking tool costs. Metrics like pass@k with large k implicitly treat tool usage as free, which becomes impractical in realistic scientific workflows where simulations or experiments consume significant time or materials. In physics simulations, the numerical parameter choices directly impact both solution quality and cost... Increasing spatial or temporal resolution improves accuracy but with cost scaling quadratically or cubically. Domain experts develop intuition for these trade-offs through experience, balancing cost against quality via informed guesses and iterative refinement. Yet without mechanisms to evaluate such cost-awareness in LLMs under current pass@k and success-rate-only metrics, we risk developing agents that achieve correctness only after numerous unaffordable trials."

"To address this gap, we present S IMUL C OST, the first benchmark designed to evaluate LLMs’ cost-awareness in tuning physics simulators. Unlike existing benchmarks that only measure success rate, S IMUL C OST additionally evaluates computational efficiency. We quantify cost by counting core operation FLOPs for each simulation instance, which correlates with wall-clock time on serial machines. Our benchmark spans 12 simulators across fluid dynamics, heat transfer, solid mechanics, and plasma physics; 11 of them admit such an analytical cost and carry the main results, while the twelfth (EPOCH, a production plasma code) is timed by wall clock and reported separately."

Main Contributions

"(1) the first benchmark measuring both success rate and computational cost for LLM-automated physics simulations; (2) an extensible toolkit with 12 simulators, 11 of them featuring platform-independent cost tracking; (3) evaluation of state-of-the-art LLMs compared against brute-force scanning and Bayesian optimization; (4) ablation studies providing practical guidance on potential improvements such as knowledge transfer, in-context learning, and reasoning effort."

Key Findings

"(1) Frontier LLMs achieve 45–62% success rates in single-round mode, dropping to 34–50% under high accuracy requirements. This indicates that LLMs’ initial guesses are unreliable and only useful for quick previews when users lack parameter intuition. (2) Multi-round mode improves success rates to 66–81%, making it necessary for high-accuracy tasks. However, LLM trial-and-error is 1.5–2.7× slower than brute-force scanning, suggesting practitioners should let LLMs invoke scanning algorithms rather than relying on internal reasoning alone. (3) Common parameters like spatial resolution are easier to tune than solver-specific ones like convergence tolerances. However, there is little correlation between individual tunable parameters even within the same type, indicating that fine-tuning on cheaper simulators is unlikely to transfer to expensive ones. (4) In-context learning improves single-round success by 17–29% but anchors models to demonstrated parameter regimes, degrading multi-round performance. This indicates naive retrieval-augmented generation may not be a complete solution."

Related Work

"LLM benchmarks for science fall into two categories: scientific knowledge evaluation and tool-using agents. The first category includes QA benchmarks such as SciBench, SciEx, and APBench... These benchmarks are limited to problems with closed-form, hand-calculatable solutions. Realistic scientific workflows require complex external tools like numerical simulations, motivating the second category: LLMs as scientific agents."

"Despite growing interest in scientific agents, most benchmarks neglect the cost of tool execution, a critical factor in real scientific workflows where tool costs (such as simulation time) dominate over LLM API costs. For example, ScienceAgentBench, DiscoveryBench, and most agentic systems track only LLM token costs. AstaBench claims 'computational cost' but reports only token costs in USD. Works such as CodePDE and SciCode combine code generation with pass@K metrics. While valuable for assessing code synthesis capabilities, these benchmarks address a different challenge than properly using existing tools, and do not consider execution cost as a metric."

"Several benchmarks address this gap by recording wall-time, e.g., Agent-Lab, MLE-Bench, MLGym, and BioML-bench. However, these measurements conflate LLM inference with tool execution time and vary across hardware, making cross-study comparisons unreliable. Auto-Bench achieves platform-independence by counting interventions as the tool cost, but this assumes uniform cost per intervention and remains in the data science domain. A related line of work adapts whether to call a tool at all, deciding by problem difficulty and trading success rate against efficiency, but the choice is binary and leaves the tool’s own cost unmeasured. CATP-LLM models cost-aware tool planning via function-as-a-service, but is limited to generic tools with relatively fixed costs per function, which breaks down for simulations where parameters like grid resolution alter cost of the same function by orders of magnitude. MetaOpenFOAM is the only prior work targeting physics simulations with cost awareness. However, it is a workflow automation framework rather than a comprehensive benchmark, and records wall-time that conflates LLM inference with solver execution."

"S IMUL C OST bridges these gaps by defining cost based on computational complexity analysis of each solver, providing platform-independent measurements with guaranteed reproducibility. We focus on parameter tuning of physics simulations where the parameters of interest fundamentally influence both simulation accuracy and computational cost."

The SimulCost Toolkit

"Our 12 simulators span fluid dynamics, solid mechanics, and plasma physics. We evaluate both single-round inference, assessing physics and numerical intuition, and multi-round inference, testing adaptive parameter optimization through solver interaction."

"Each solver presents LLMs with parameter tuning tasks: given a physics-based simulation scenario, the LLM must select parameter values that satisfy accuracy requirements while minimizing computational cost. In reality, simulators usually require tuning multiple parameters for a successful execution. However, domain experts usually isolate potential issues upon seeing unsuccessful simulation runs and tune the corresponding parameters one by one. We adopt this pattern and constrain the LLM to tune individual parameters while fixing others to reasonable (but non-optimal) values chosen by rule-of-thumb practices (e.g., CFL= 0.25 for explicit time-stepping). This isolation avoids the complexity of multi-parameter optimization and enables a meaningful grid search (scan) baseline. For each solver, we define the tool cost using computational complexity analysis by counting dominant operations FLOPs, which strongly correlates with wall time on a single-core processor."

Dataset Curation

"Our curation pipeline has four phases: (1) Solver development - developing new or adapting existing solvers, (2) Reference solution search - finding reference parameters for each task using brute-force search, (3) Variation design - scaling up task diversity by combining accuracy thresholds (low/medium/high), initial/boundary conditions, environments, and non-target parameter settings, and (4) Filtering - since experts define the accuracy thresholds consistently for all combinations, some may fail to find a solution that meets the threshold within the maximum scan iterations. We filter these infeasible tasks post-hoc to avoid unnecessarily high computational time for later LLM evaluations."

"All solvers are sourced from textbooks, online repositories, and research papers. Original scripts typically contain bugs or lack cost-related metadata, requiring expert debugging and adaptation for quality assurance, cost calculation, and consistent API design."

Inference Modes

"Expert parameter tuning in physics-based simulations requires two distinct capabilities: (1) physical and numerical intuition for initial parameter estimation and (2) iterative adjustment through interactions with solvers. We design two inference modes to evaluate these skills, measuring both success rate and cost efficiency:

Single-Round Inference requires single-attempt parameter selection using LLM’s prior knowledge to find the optimal parameter balancing accuracy and cost. Since the true optimal in continuous parameter space is unknown, we set the reference as the near-optimal solution with near-minimal cost that meets the accuracy threshold found by scan algorithm.

Multi-Round Inference allows up to 10 trials. Human expertise usually only has patience for 5–10 trials before resorting to systematic sweeps, and our scan baseline uses approximately 20 grid points, so 10 trials gives LLMs half the budget of exhaustive search. Each trial’s simulation feedback includes: a) convergence status, b) RMSE between the current and a finer resolution to judge convergence, and c) accumulated cost is appended to the conversation list. LLMs can terminate early if they deem a solution satisfactory."

Evaluation Metrics

"Success Rate. Success Si ∈ 0, 1 for task i indicates whether the task-dependent distance between the LLM-proposed simulation output and the reference output (found by Algorithms 1 or 2) falls within the accuracy threshold."

"Efficiency. Efficiency measures how well the LLM’s cost compares to the reference cost: Ei = (Cibf / Cisr,mr) × Si, where Cibf is the brute-force reference cost, Cisr,mr is the LLM’s cost, and superscript 'sr' or 'mr' denotes single-round or multi-round mode respectively. For single-round, Cibf,sr is the near minimum-cost solution; for multi-round, Cibf,mr is the accumulated scan cost."

"For aggregated results, we report arithmetic mean for success rate and geometric mean for efficiency over successful samples only (Ei > 0). The geometric mean is standard practice for aggregating performance ratios and speedups: it is scale-invariant, handles values spanning multiple orders of magnitude without being dominated by outliers, and treats multiplicative relationships symmetrically. Failed tasks (Ei = 0) are excluded from efficiency aggregation."

Experiments

"We evaluate state-of-the-art LLMs including GPT-5-2025-08-07, Claude-3.7-Sonnet-2025-02-19, GPT-OSS-120B, Llama-3-70B-Instruct, and Qwen3-32B across the 11 solvers with analytical costs spanning fluid dynamics, solid mechanics, and plasma physics. Our evaluation encompasses both single-round and multi-round inference modes at three accuracy levels (low, medium, high). Temperature is set to 0.0 for all models except GPT-5, which uses its default setting."

Main Results

"In single-round mode, success rates vary substantially across models (45–62%), with GPT-5 leading at 61.5%. Even this best rate falls short of practical reliability, requiring manual intervention every 2–3 simulations. Multi-round inference substantially improves all models, with GPT-5 reaching 80.8% and GPT-OSS-120B 77.1%, and even Llama-3-70B reaching 71%."

Regarding performance across different accuracy levels, single-round success drops sharply as accuracy requirements tighten (64% → 39%), while multi-round remains stable (73% → 71%).

"Efficiency interpretation differs by mode: in single-round, efficiency >1.0 means the LLM beat the optimal solution’s cost. In multi-round, the reference is cumulative brute-force cost, so efficiency ≈1.0 merely matches naive search. In single-round, all models achieve 0.16–0.52 efficiency, using 2–6× optimal compute. In multi-round, only GPT-OSS-120B reaches 0.86 (at low accuracy), while other models cluster around 0.3–0.7, taking 1.5–2.7× brute-force cost on average."

Single-Round vs. Multi-Round Comparison

"We observe that the success rate gain is most pronounced at high accuracy levels, with a mean improvement of +30.5% across models (p < 0.001), precisely where single-round struggles most. Lower accuracy levels also benefit, but to a lesser extent. This asymmetry has a clear explanation: higher accuracy requirements narrow the acceptable parameter range, making 'lucky guesses' increasingly unlikely. Multi-round mode compensates through trial-and-error, making it necessary for high-accuracy tasks."

Task Group Analysis

"Success rate varies substantially across parameter groups in single-round mode (15–68%). Spatial and Tolerance parameters cluster at the high end, likely benefiting from predictable cost-accuracy trade-offs in pre-training data, while Misc parameters show the widest variation due to their solver-specific nature. Multi-round mode compresses this variation (all groups: 56–79%), with Misc parameters gaining most (+28%). This pattern suggests that iterative exploration compensates for missing prior knowledge, benefiting uncommon parameters the most."

"Efficiency patterns reveal a paradox: in single-round, Misc parameters (the hardest to solve) achieve near-optimal efficiency when successful, while Spatial parameters lag far behind. This suggests LLMs can find runnable values for common parameters like grid resolution but lack cost intuition, tending toward 'safer' values that work but waste compute."

"We analyzed within- vs. between-group task correlations to assess knowledge transfer potential. Neither single-round nor multi-round showed significant within-group correlation advantages (p=0.24 and p=0.88, respectively), implying task difficulty is parameter-specific rather than type-driven. This limits the viability of fine-tuning on cheap simulators to improve performance on expensive ones."

In-Context Learning

"ICL improves single-round success but degrades multi-round performance. This pattern holds across all variants, suggesting that examples help narrow initial guess ranges but limit exploration, steering models to shown parameter regimes."

"Comparing variants in single-round mode: Mixed-Accuracy ICL achieves the largest gains in both success (+28.7%) and efficiency (1.74×), likely because diverse examples span a wider parameter range. Cost-Ignorant ICL shows comparable success gains but minimal efficiency improvement, suggesting that cost information in examples is critical for efficiency."

Bayesian Optimization Baseline

"BO achieves comparable aggregate success rates to LLMs but shows higher inter-solver variance. Efficiency-wise, LLMs have an obvious advantage, especially at low accuracy requirements (2.13 vs. 1.05). We found that BO’s exploration strategy tends to choose extreme values early to maximize information gain. However, this triggers early stopping if it selects the 'finer' side of extreme bounds, resulting in high cumulative cost. In contrast, LLMs leverage physics intuition from pre-training to make more informed initial guesses, which is especially helpful at low accuracy requirements where thresholds are more lenient."

Additional Results

"Reasoning Effort: GPT-5’s reasoning effort parameter shows no significant overall impact, as increased reasoning does not lead to better parameter selection. Failure Modes: We identify five recurring failure patterns (false positives, blind exploration, instruction misunderstanding, prior bias, and conservative strategy) to guide future improvements. These patterns reveal a tension between cost-awareness and correctness: LLMs either skip intermediate resolutions to save cost and land on non-converged solutions (blind exploration), or choose unnecessarily fine resolutions 'to be safe' and then make tiny adjustments that hinder exploration (conservative strategy). Additional Baselines: Classical optimization methods (Bisection, Nelder-Mead) achieve similar success rates to brute-force at comparable cost, justifying our baseline choice. Robustness: Prompt sensitivity analysis (25 variants) and unconditional cost metrics leave our conclusions unchanged. Tool-Augmented Tuning: When LLMs provide initial guesses for search algorithms, success improves (+8.6%) but cost doubles due to conservative parameter choices. Wall-Clock-Cost Solver: EPOCH, whose cost cannot be counted in FLOPs, is evaluated on its own and excluded from every aggregate above."

Limitations

"Our benchmark restricts LLMs to text-based solver interactions without access to auxiliary tools. Real-world agents might leverage log parsing or custom timeout logic, but this constraint ensures fair comparison: many LLMs do not yet support full agentic mode reliably, and the choice of framework introduces confounding variables."

"Additionally, we isolate single-parameter tuning rather than joint multi-parameter optimization. Baseline tractability drives this choice: grid search complexity grows as O(nk) where n is the number of choices per parameter and k is the number of parameters. With n = 20 grid points, single-parameter search requires 20 evaluations, while 3-parameter joint search requires 203 = 8,000, making exhaustive reference solutions prohibitively expensive."

Conclusions and Future Directions

"We introduced S IMUL C OST, the first benchmark for evaluating cost-aware parameter tuning capabilities of LLMs in physics simulations. Our main evaluation spans the 11 physics simulators with analytical costs and 5 state-of-the-art LLMs; EPOCH is reported separately. We also open-source S IMUL C OST upon publication, including the static benchmark and extensible toolkit."

Practical recommendations

"(1) LLMs’ initial guesses are generally unreliable (45–62% success) and only useful for quick previews when users lack parameter intuition, accuracy requirements are low, and optimal cost efficiency is not needed. (2) For high-accuracy tasks or finding a guaranteed cost-efficient configuration, multi-round mode becomes necessary (66–81% success), but practitioners should let LLMs invoke scanning algorithms rather than relying on internal reasoning alone, as the latter is 1.5–2.7× slower. (3) Fine-tuning on cheaper simulators is unlikely to transfer to expensive ones due to the lack of cross-parameter correlation, even within the same parameter type. (4) ICL improves single-round performance but anchors models to demonstrated regimes, degrading multi-round exploration. Hence, naive RAG may not be a complete solution. (5) Using examples from a diverse range of accuracy requirements helps exploration, and including cost information helps optimize cost efficiency. (6) LLM-initialized search should be steered toward intermediate parameter values: conservative initial guesses undershoot the optimum by 50–80%, incurring 5–14× cost overhead."

Future directions

"(1) Tool-augmented tuning: equipping LLMs with timeout-based early stopping, callable search algorithms, and multi-modal feedback such as field visualizations to enable richer decision-making. (2) Human-in-the-loop evaluation: user studies measuring how LLM-suggested parameters accelerate expert workflows, other than autonomously tuning, would validate real-world utility. (3) Cost-aware post-training: developing fine-tuning strategies that explicitly optimize for accuracy and computational efficiency. (4) Multi-parameter optimization: enabling LLMs to jointly tune interdependent parameters with adaptive sampling approaches. (5) Parallel computing: extending cost tracking to parallel architectures with analytical complexity models that account for overhead and delays, or careful, reproducible wall-time measurements. (6) Non-monotonic parameter handling: for parameters without clear cost-accuracy monotonicity, experts either draw on mathematical or physics priors or inspect visualizations after the fact for abnormal behavior such as oscillations near shock fronts."

Based on the paper's findings, here are specific improvements I can implement in AI systems:

Improvement: Replace pass@k and success-rate-only metrics with a dual metric system that tracks both success rate and computational efficiency (cost ratio vs. brute-force baseline).

What the improved system can do: Evaluate LLM agents on physics simulation tasks while penalizing excessive computational resource usage, preventing agents from succeeding through wasteful trial-and-error.

Improvement: Implement a hybrid decision mechanism that automatically selects between single-round and multi-round modes based on task accuracy requirements and parameter type.

Improvement: Add explicit cost-minimization instructions and include cost information in any in-context examples.

Improvement: Equip the agent with callable brute-force scanning algorithms rather than relying on internal reasoning for multi-round refinement.

Improvement: Implement a diversity check on retrieved examples to prevent anchoring to demonstrated parameter regimes.

Improvement: Add a post-generation validation step that checks if proposed parameters are within a safe intermediate range rather than extreme values.

Improvement: Implement different strategies for different parameter groups (Spatial, Temporal, Tolerance, Misc).

Improvement: Add cumulative cost monitoring that triggers early termination when costs exceed a threshold relative to expected brute-force cost.

Improvement: Implement a feedback analysis module that extracts actionable patterns from simulation results before proposing next parameters.

Improvement: Add a task-similarity check before applying fine-tuned knowledge from one simulator to another.

Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,643 single-round (initial guess) and 2,304 multi-round (adjustment by trial-and-error) tasks across 11 simulators from fluid dynamics, solid mechanics, and plasma physics, whose costs are analytically defined and platform-independent. A twelfth simulator, a production plasma code measurable only by wall clock, is reported separately. Frontier LLMs achieve 45-62% success rates in single-round mode, dropping to 34-50% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 66-81%, but LLMs are 1.5-2.7x slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source SimulCost as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench