ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

arXiv:2604.15994 · cs.AI · Submitted 2026-04-17 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams".

Jane: The paper was written by the authors from University1 and Company2.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Jane: So, if Segment one was about *what* the benchmark is, this segment dives into *how* it works and what that means for MLLMs. The paper details specific tasks—things like identifying reactants, products, and key intermediate steps from a reaction sequence.

Tom: It’s not just recognizing the start and end materials, right? They're forcing the models to track every single step along the way, which is what makes it so hard for current AI systems.

Lu: And this tracking ability is crucial because chemical reactions are often multi-step processes, meaning the model needs to maintain a consistent internal state of matter and energy through time, which mimics real scientific intuition.

Jane: Right! They summarize that models must handle entire pathways, not just isolated single reactions. This requires them to understand the causality—that step B only happens *because* of what happened in step A.

Meng: From an engineering standpoint, having a clear summary of these tasks is gold because it allows us to build targeted testing harnesses. Instead of vague performance metrics, we can measure specific failure modes, like when the model fails to track stoichiometry across multiple steps.

Lalam: And that ability to pinpoint failure modes is what truly accelerates scientific discovery; it tells the human researchers exactly where their current understanding or toolset breaks down, guiding them toward the next breakthrough in chemical engineering.

Tom: So, Jane, if I understand correctly, they are moving beyond simple data extraction and into temporal reasoning within chemistry.

Jane: Exactly. They show that success isn't just knowing the inputs and outputs; it’s articulating the *process* of transformation using all the visual cues provided in the diagram.

Lu: It’s a test of deep structural coherence, Jane, and that goes far beyond merely calling out chemical formulas; it requires understanding chemical logic.

Meng: If we could build a tool based on this summary, I think our immediate practical application would be automating preliminary feasibility studies for industrial chemistry—giving engineers an instant assessment of whether a proposed reaction sequence is chemically plausible.

Lalam: And imagine the cultural shift: instead of relying on expensive and time-consuming expert consultation for initial pathway planning, AI could provide a highly reliable first pass, democratizing access to complex chemical process design.

Tom: Wow, so we've moved from just understanding structure to understanding the entire narrative of transformation. This leads us perfectly into how this benchmark is helping the field improve.

Improvements: Tom: We talked about how ReactBench summarizes the task, but this paper also discusses improvements—how can AI models get *better* at these kinds of complex reasoning tasks?

Jane: They suggest that simply feeding the model a massive amount of data isn't enough; the improvement needs to come from structuring the data and

Paper discussion segment 3: Tom: We’ve spent a lot of time looking at how ReactBench exposes that major gap between an AI’s ability to see a single chemical entity and its inability to grasp the entire reaction process.

Jane: That's exactly what it is; the models can identify a specific reactant, but they struggle to piece together the whole story of transformation across multiple steps.

Lu: It moves us past simple recognition toward needing true hierarchical abstraction, which is fundamentally about understanding how a complex network operates as a whole system.

Meng: From an engineering view, this means that we are no longer just looking for better OCR; we need to build systems capable of simulating the entire chemical pathway based on visual logic.

Lalam: I think the biggest cultural shift will be that AI could provide instant, comprehensive initial pathway planning, democratizing access to complex chemical process design.

Tom: That’s a huge leap—moving from just theoretical possibility to real-world industrial application. It feels like we're going from understanding pieces to understanding the entire narrative.

Jane: Right, and it's not just about speed; it’ also about correctness, ensuring that the AI understands the causal link between what happens in step one and what enables step five.

Lu: That structural coherence is key, which is why this benchmark forces a deeper level of graph-theoretic reasoning onto these large models.

Meng: If we can automate that pathway tracing reliably, we can dramatically reduce the time spent on early-stage synthesis and resource planning for industrial chemists.

Lalam: And by improving our AI's ability to interpret complex structures, we are helping the next generation of scientists move faster than ever before.

Tom: It’s a monumental step forward, moving from just understanding structure to understanding the entire narrative of transformation. It makes me wonder what other scientific domains might benefit from this kind of topological reasoning.

Conclusion: Tom: So we’ve seen how ReactBench puts these MLLMs to the test by looking at their ability to trace chemical reaction pathways, and what that reveals about their current limits in structural thinking.

Jane: It's a powerful tool for understanding exactly where AI gets stuck—it sees the individual parts but can't always put them back together into a cohesive whole.

Lu: I think the biggest implication is that we need to move away from just training models on huge amounts of text, and start building architectures that inherently understand graph theory.

Meng: From my side, it means our next generation of AI tools can be more reliable for automating complex chemical analysis because they won’t just guess at the steps.

Lalam: I hope this work inspires a culture where we trust AI not just as a quick answer generator, but as a rigorous partner in scientific discovery.

Tom: That sounds like the right way to frame it—a collaborator instead of just a calculator. It's interesting that even with massive scaling, these foundational reasoning gaps persist.

Jane: Exactly, so we’ve seen that while they excel at localized perception, they fail to integrate those local elements into a coherent global structure.

Lu: This is where the theory meets reality; the models are stuck in local detail when the chemistry demands a holistic view.

Meng: We can't ignore this bottleneck, because if we’re trying to industrialize chemical synthesis using AI, reliable pathway tracing is non-negotiable.

Lalam: The goal for us is to ensure that all future generations of AI models are capable of grasping the full narrative of transformation in complex systems.

Tom: That's a big goal, but it’ worth remembering that the this foundational work on ReactBench really shines by pointing out exactly where our current MLLMs are struggling with topological reasoning.

Jane: It serves as a clear map for researchers to see what needs to be fixed in future models.

Lu: And I think seeing the failure modes is often more important than finding a solution, because it tells us precisely where the challenges lie.

Meng: It’s all about identifying that exact structural weakness, and that's exactly what this paper achieved.

Lalam: Let's hope this study sets a path for a truly holistic approach to how we use AI in science.

Tom: We will certainly be keeping an eye on how the community responds to "ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams."

University1 · Company2

cs.AI

Submitted: 2026-04-17

Updated: 2026-08-31

Importance score: 84/100

The gist: This paper introduces ReactBench, a specialized benchmark designed to "diagnose structural reasoning limitations" in Multimodal Large Language Models (MLLMs) through the use of complex chemical

Key concepts

ReactBench
ReactBench is a benchmark designed to test the ability of Multimodal Large Language Models (MLLMs) to perform topological reasoning on chemical reaction diagrams. It moves beyond simple data extraction by requiring models to track the entire transformation process, rather than just identifying initial reactants and final products.
Topological Reasoning
This involves understanding a complex system as a whole, not just its isolated components. In this context, it means grasping the complete narrative of transformation—understanding how one step enables subsequent steps—to achieve structural coherence in chemical processes.
Temporal Reasoning
This is the ability to track causality and maintain consistency across multiple stages of a process. It requires models to understand that a specific intermediate step (B) only occurs because of what happened in the previous step (A), allowing them to follow the entire reaction pathway over time.

Terminology

Summary

This paper introduces ReactBench, a specialized benchmark designed to diagnose structural reasoning limitations in Multimodal Large Language Models (MLLMs) through the use of complex chemical reaction diagrams. It addresses a critical gap in current research, where models excel at recognizing individual visual elements but fail when faced with complex topological structures involving branching paths, converging flows, and cyclic dependencies.

The ReactBench Dataset

ReactBench is constructed from a collection of over 1,300 unique real-world chemical reaction diagrams curated from academic literature and patent databases, specifically spanning the period from 1996 to 2016. To ensure maximum diversity in layout variability and notation styles, the researchers employed a strict selection strategy where every image is sourced from a distinct article. The resulting ChemReaction dataset comprises 1,618 expert-annotated QA pairs that span diverse structural complexities. To capture the full spectrum of visual complexity, the researchers stratified the diagrams into four categories representing different reaction topologies:

  • Single-line: Linear pathways with unidirectional flows.

  • Multiple-lines: Parallel or branched mechanisms with competing intermediates.

  • Tree: Hierarchical branching with divergent synthesis routes.

  • Graph: Cyclic networks requiring non-linear reasoning.

Hierarchical Task Dimensions

To identify exactly where model capabilities degrade along the perception-to-reasoning spectrum, ReactBench evaluates MLLMs across four hierarchical task dimensions:

  1. Spatial Element Localization: This task evaluates the capacity for fine-grained spatial reasoning and object localization within structured diagrams, requiring models to identify node types and edge properties based on topological position.

  2. Topological Information Extraction: This dimension assesses the ability to perform structured information extraction guided by topological context, challenging models to maintain correspondence between visual tokens and semantic roles.

  3. Pathway Connectivity Tracing: This probes the model's grasp of connectivity and path traversal in reaction networks, demanding graph-theoretic reasoning to reconstruct complete paths.

  4. Structural Topology Reasoning: This focuses on the ability to classify and analyze the overall topological structure of diagrams, such as detecting cycles or identifying terminal nodes.

Performance Analysis and Findings

Extensive evaluation across 17 MLLMs reveals a significant performance dichotomy between different task types. While models are highly proficient at anchor-based tasks—with accuracy exceeding 80% for information extraction and path tracing—their performance drops sharply on tasks requiring holistic structural reasoning. The researchers found a performance gap exceeding 30% between these two categories, indicating that structural complexity is a primary determinant of task difficulty. For example, while top-tier models like Qwen-VL-MAX achieve 90.10% accuracy in extraction, they fall to only 32.18% in reasoning tasks.

Controlled ablations demonstrate that this bottleneck is rooted in reasoning rather than perception. Even when models are provided with error-free structural information as External Knowledge (EK) or when the visual input is replaced by a linearized text-only JSON representation, performance on holistic tasks remains critically low. This suggests that current architectures lack the inductive bias necessary for global structural reasoning and suffer from a core deficit in hierarchical abstraction. Models often fail to move from local feature recognition to global process understanding, frequently disregarding explicit topological rules in favor of learned heuristics or superficial numerical sequences.

Improvements for AI systems

The Improvement: Replace or augment current Vision Transformer (ViT) backbones with a dual-stream architecture that incorporates a Graph Neural Network (GNN) or Graph Transformer layer specifically designed to process topological primitives (nodes and edges) extracted from the visual field. This involves an intermediate Structural Parsing Module that converts visual connectivity (arrows, branching points, convergence flows) into a mathematical graph representation before it reaches the LLM reasoning core.

What the improved AI can do:

  • Perform Graph-Theoretic Operations: The system will be able to mathematically identify terminal nodes (sinks), calculate the shortest path between two compounds in a complex network, and detect cyclic dependencies (loops) with high precision.

  • Execute Global Topology Classification: It will correctly classify reaction schemes as Single-line, Tree, Multiple-line, or Graph structures by analyzing the connectivity matrix rather than relying on superficial visual heuristics.

The Improvement: Implement a specialized pre-training phase using Topological Reasoning Tasks rather than standard Image-Text alignment. Instead of just image of molecule SMILES string, the training data should consist of image of connectivity graph properties (e.g., degree of nodes, path length, branching factor). This forces the model to learn the logic of connections as a fundamental modality.

The Improvement: Develop an attention mechanism that can dynamically shift its weight between Semantic Attention (focusing on molecular structures and text) and Structural Attention (focusing on arrows and directional flow). This mechanism would be triggered by the query type: if the query is What is the yield?, it weights semantic tokens; if the query is How many end products?, it weights topological edges.

The Improvement: Integrate a neuro-symbolic reasoning loop where the MLLM's visual output is validated against a symbolic graph engine. Before finalizing an answer regarding connectivity or counting, the model must check its reasoning against a formal graph representation of the diagram it just perceived.

Sources

Related papers