One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning

arXiv:2608.30952 · cs.LG, cs.CL · Submitted 2026-08-31 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning".

Jane: The paper was written by Armin Dariani, Sifan Wu, Bang Liu and Entao Yang from Université de Montréal and Mila - Quebec Artificial Intelligence Institute and Innovation Campus Delaware, Air Liquide (Air Liquide).

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: We've set up that dynamic view, but let’s drill down into what this "rollout protocol" is and how it contrasts with previous methods. The authors are using a single policy that interleaves reasoning with live tool execution in one generation of the AI model.

Jane: I want to make sure our listeners understand that instead of having two separate models—a planner and an executor—the rollout protocol tightly weaves thinking and action together in a single pass. It's a continuous loop where the AI thinks, calls a tool, gets a real result from that tool server, and then decides what to do next based on exactly what it just saw.

Lu: This is crucial because it means the agent doesn't have to guess or predict how things will work; it sees the actual output from the tool server immediately after making its call. That instant, empirical feedback allows for incredibly precise reasoning about complex chemical structures and their properties that are needed for subsequent steps in a real-time flow.

Meng: I think that real-time feedback loop is a massive game changer for practical implementation, too. We aren't relying on simulated outcomes or assumptions; we're using the actual execution of tools like those found in ChemCrow, which is much more reliable for engineering tasks that require hard numbers and precise chemical data.

Lalam: For us, this means AI can move beyond just being a static predictive engine and start becoming an active participant in the scientific workflow itself. It’s like giving the AI a hands-on toolkit to conduct experiments right within its own thought process, which is a big cultural shift in how we utilize these systems.

Tom: That continuous loop of real feedback replacing the abstract search—it’s clear that "One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning" is proposing a much more dynamic and responsive approach than what we've seen before. How does this dynamic approach translate into actual measurable results?

Improvements: Tom: We've established the mechanism, but the real question for our listeners is how well does this single-agent system actually work in practice? The paper claims significant improvements over their previous state-of-the-art method, CheMatAgent.

Jane: They are reporting concrete gains across several metrics on the ChemToolBench dataset that are very impressive. Specifically, they improved Tool F1 by five point five percent and Return F1 by nine point six percent on Qwen-two point five-7B models, which is a massive leap for a single agent to achieve this level of accuracy in both selecting the right tool and making the argument precision across multi-step problems.

Lu: This performance jump confirms the hypothesis that the previous search mechanism wasn't just an overhead; it seems like "One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning" shows us that a single policy is actually sufficient to handle all the complexity of those long, multi-step chemical problems.

Meng: From an engineering standpoint, those F1 improvements are exactly what translate into better performance at scale. If the AI can select tools more accurately and use their specific outputs correctly, we can build much more robust and reliable automated chemistry pipelines for industry that were previously too fragile.

Lalam: The fact that these gains are seen across different underlying models suggests a generalizable improvement in how AI learns tool usage. This is exciting for the wider implications of how we design intelligent systems, showing us that these new learning techniques are improving fundamental reliability itself.

Tom: So, we have the data points and the technical improvements; it's clear that this single policy approach is proving to be highly effective. But before we move on, I want to hear your quick thoughts on what these gains mean for our listeners as we look at the results from "One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning.

Improvements: Tom: We’ve seen the technical improvements, but it's important to talk about what this all means in terms practical application. The paper isn't just showing a theoretical gain; they are demonstrating how well this single agent performs on real-world chemistry questions.

Jane: The results are remarkably high for a single-agent approach; achieving that nine point six percent improvement in Return F1 means the AI is not only picking the right tool but successfully chaining those calls together so that its final answer is chemically valid and accurate, which is a huge step beyond just selecting tools and avoiding errors.

Lu: I think this confirms that by moving away from complex tree searches, we are allowing the AI to focus on what it does best—reasoning through a continuous flow of actual data—rather than trying to manage an abstract, multi-layered search space. It’s about focusing on the immediate reality of discovery in chemistry.

Meng: The practical impact here is that if we can trust the tool selection and argument use this much more reliably, we can build much smaller, faster models that still perform at high levels of accuracy for chemistry tasks. That reduces hardware costs significantly across many different industries.

Lalam: This suggests a future where AI becomes a highly reliable co-pilot in scientific research, allowing us to move past initial automated hypothesis testing and toward complex design and optimization cycles in chemical synthesis. It elevates the role of AI from simple calculator to active partner in science.

Tom: We have the data points showing both that technical superiority and practical reliability; it's clear that "One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning" is proving to be highly effective. But before we move on, I want to hear your quick thoughts on what this means for our listeners as we discuss the real-world use cases of this paper.

Conclusion: Tom: We’ve spent the last few minutes breaking down how this single-agent system works, and it’s truly a paradigm shift in how we structure AI agents. I think we can all agree that the heavy machinery of tree search is being replaced by something much more elegant and powerful.

Jane: It certainly feels like a major step toward making AI agents far more competent in scientific domains because "One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning" shows they can handle those complex, dependent calls without needing an entire, slow planning layer.

Lu: I’m thrilled to see this, and I think it opens up a world where we can tackle even more intricate chemical syntheses because of the improved tool precision. We're looking at a future where AI guides discovery in ways we haven't even imagined yet.

Meng: My final thought is that this translates directly to much lower computational overhead for future systems, allowing us to deploy these powerful chemistry agents on more varied and affordable hardware for the world’s engineers.

Lalam: I hope that AI can continue evolving in this direction, allowing it to not just solve problems but to assist humanity in understanding the world with greater depth and efficiency. We must see its advancement as a whole, not just a series of incremental steps.

Tom: Thank you all for joining us today! That's our final look at the findings of "One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning." It’s been a fantastic discussion, and we hope you enjoyed hearing how this AI is changing the landscape of chemistry.

Jane: Until next time, everyone!

Armin Dariani, Sifan Wu, Bang Liu, Entao Yang

Université de Montréal · Mila - Quebec Artificial Intelligence Institute · Innovation Campus Delaware, Air Liquide (Air Liquide)

cs.LG, cs.CL

Submitted: 2026-08-31

Updated: 2026-08-31

Code: https://github.com/ur-whitelab/chemcrow-public

Importance score: 84/100

The gist: This paper presents a novel approach to chemistry tool learning for large language models, challenging the necessity of complex, multi-model tree search architectures.

Key concepts

Rollout Protocol: This protocol allows the AI model to interleave reasoning with live tool execution in one pass. Instead of using separate planning and executing models, it continuously thinks, calls a tool, receives a real result, and decides the next action based on that immediate feedback.
Single Policy Approach: The authors propose using one policy to handle all complexity in multi-step chemical problems. This method is highly effective because it focuses on reasoning through a continuous flow of actual data rather than managing an abstract, multi-layered search space.
Reinforcement Learning (RL): A machine learning technique where an agent learns by interacting with an environment. In this context, the AI receives 'real-time feedback' from chemical tools after making a call, allowing it to refine its actions and improve its performance.

Terminology

Reinforcement Learning — 21× in this paper · explained in 4 episodes
Benchmark — 17× in this paper · explained in 2 episodes
GRPO — 6× in this paper · explained in 6 episodes
Accuracy — 5× in this paper · explained in 1 episode
Fine-Tuning — 3× in this paper · explained in 5 episodes
AI Agents — 2× in this paper · explained in 2 episodes
Group Relative Policy Optimization — 2× in this paper
LLM judge — 2× in this paper · explained in 2 episodes
Process Reward Model — 2× in this paper
Quantitative Estimate of Druglikeness — 2× in this paper
Verifiable Rewards — 2× in this paper · explained in 2 episodes
Ablation — 1× in this paper · explained in 1 episode
Computational overhead — 1× in this paper · explained in 2 episodes
Decode — 1× in this paper · explained in 2 episodes
F1 Score — 1× in this paper · explained in 3 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 5 episodes
Process Reward Model (PRM) — 1× in this paper · explained in 3 episodes
Reward Hacking — 1× in this paper · explained in 8 episodes
Trace — 1× in this paper · explained in 2 episodes

Summary

This paper presents a novel approach to chemistry tool learning for large language models, challenging the necessity of complex, multi-model tree search architectures. By demonstrating that a single-agent reinforcement learning policy can outperform hierarchical evolutionary Monte-Carlo Tree Search (HE-MCTS), the authors provide a more efficient and effective method for agents to navigate specialized chemical tasks involving complex tool selection, structured argument generation, and long-horizon reasoning.

The challenge of chemistry tool use

Chemistry questions demand exact computation and database lookups that language models cannot supply from their parameters, necessitating the use of external tools. This creates a three-part problem: the agent must select the right tool from a large pool, fill it with correctly typed arguments, and chain calls so that each consumes the outputs of the last.

The paper identifies several specific difficulties in this domain:

Chemistry tools are highly specialized and challenging for general-purpose LLMs to reason.
Tools require structured inputs with strict scientific validity and computational precision, such as SMILES strings or chemical formulas, where errors can invalidate execution or lead to scientifically incorrect results.
Problems often require long-horizon tool interaction, meaning early mistakes propagate through the execution trajectory and compound into downstream failures.

How it works

The authors propose a single-agent rollout protocol where a single policy interleaves reasoning, tool calls, and returns in one left-to-right generation. This approach dispenses with the execution model, the PRM and ORM critics, and the tree search used by previous state-of-the-art systems. Instead of a complex hierarchical search, the model alternates thinking and tool calls inside a single generation, reading real execution results to inform subsequent steps.

Training is conducted in two stages. First, the policy undergoes a supervised warm-up (SFT) to teach the output format and provide a first approximation of correct tool selection. This is followed by reinforcement learning using Group Relative Policy Optimization (GRPO) against a verifiable reward read directly off the gold call chain.

A programmatic reward framework

To avoid the expense of learned critics or LLM judges, the authors develop a systematic reward framework for multi-tool calling using programmatic rewards that capture answer correctness, tool selection, tool-argument accuracy, and their combinations. These rewards are cheap, deterministic, and reproducible.

The framework includes several key reward variants:

Rans (value inclusion): The fraction of ground-truth return values that appear verbatim in the final answer.
Rtool (tool set): The F1 score over the set of tool names.
Rcall (tool and arguments): An F1 variant where a call is only successful if its arguments also match the ground truth.
Rhyb (hybrid): An equal mix of tool and answer rewards.

Key findings and performance

The single-policy model consistently outperforms existing SOTA approaches across multiple metrics on the ChemToolBench benchmark. On the Qwen-2.5-7B backbone, the model improves Tool F1 by 5.5% and Return F1 by 9.6% compared to the strongest search configuration. On Llama-3.1-8B, it improves Tool F1 by 3.7% and Return F1 by 3.9%. These results are achieved at one model invocation per question, whereas the cost of previous tree-search methods grows with the size of the tree.

The model also demonstrates self-repair from execution feedback. Because the protocol executes tools for real in training as well as at test, the policy can read error strings returned by the tool server. When a mistake occurs, the model re-issues the call with the fault corrected instead of continuing as though it had succeeded, a capability that is not learned through supervision since gold chains only contain successful calls.

Improvements for AI systems

1. Transition from Multi-Model Hierarchical Architectures to Single-Policy Interleaved Rollout Architectures

Improved System Capability: The system can execute complex, multi-step scientific workflows (e.g., retrieving a molecular CID, converting it to SMILES, and then calculating LogP/QED) in a single, left-to-right generation. This eliminates the need for separate planning, execution, and critic models, drastically reducing inference latency and computational costs while maintaining high reasoning accuracy.

2. Implementation of GRPO-based Reinforcement Learning with Programmatic Verifiable Rewards

Improved System Capability: The system achieves high precision in tool selection and parameterization by optimizing against deterministic, code-based rewards (matching tool names, argument types, and return values) rather than unreliable learned reward models (PRM/ORM) or LLM judges. This prevents reward hacking and ensures that the agent's reasoning is grounded in verifiable scientific truth.

3. Integration of Execution-Feedback-Driven Self-Repair

Improved System Capability: The system can autonomously detect and correct errors in real-time. If a tool call fails due to a syntax error (e.g., a typo in an argument name like compound CID instead of compound CID), the agent reads the error string from the tool server and re-issues a corrected call within the same reasoning trajectory, preventing the propagation of errors in long-horizon tasks.

4. Deployment of Multi-Faceted List-Semantic Reward Shaping

Improved System Capability: The system avoids tool spamming (invoking all available tools to increase recall) and padding (adding unnecessary calls) by utilizing list-based F1 rewards. This forces the agent to produce highly efficient, minimal, and scientifically accurate tool chains where every call is necessary and every argument is precisely formatted.