Transformers Discover Molecular Structure Without Graph Priors
cs.LG, cond-mat.mtrl-sci, physics.chem-ph, q-bio.BM
Submitted: 2025-10-02
Updated: 2026-09-24
Project page: https://tkreiman.github.io
License: http://creativecommons.org/licenses/by/4.0/
The gist: Computational simulations play a central role in scientific discovery, and machine learning (ML) has emerged as a promising alternative to traditional physics-based modeling.
Terminology
Abstract
Computational simulations play a central role in scientific discovery, and machine learning (ML) has emerged as a promising alternative to traditional physics-based modeling. However, scientific modeling requires physically meaningful predictions, raising a fundamental question for data-driven methods: to what extent can physical inductive biases - that is, prior assumptions about the structure of the physical world - emerge by learning from data alone? In atomistic modeling, for example, ML architectures have historically embedded strong physical inductive biases - such as geometric locality and graph structure - based on the assumption that these priors are necessary for physical predictions. We systematically develop an understanding of how physical patterns can alternatively be discovered directly from data by training a model without domain-specific priors, including any manually defined atomistic pairwise interactions. We find that the model autonomously recovers key physical structure, such as learned interatomic interaction strengths that mirror classical electrostatics and interaction cutoffs consistent with traditional physical models. We further demonstrate predictable neural scaling law behavior with increased data and compute, and find accuracy on certain metrics competitive with physics-informed architectures. Our results clarify the boundary between engineered inductive bias and learnable physical structure, suggesting that general-purpose architectures can serve as principled baselines for scientific modeling, with explicit priors introduced only when empirically necessary.
Sources
- Open Materials 2024 (OMat24) Inorganic Materials Dataset and Models
- A foundation model for atomistic materials chemistry
- Graph Neural Networks Use Graphs When They Shouldn't
- EGraFFBench: Evaluation of Equivariant Graph Neural Network Force Fields for Atomistic Simulations
- Language Models are Few-Shot Learners
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- FAENet: Frame Averaging Equivariant GNN for Materials Modeling
- How simple can you go? An off-the-shelf transformer approach to molecular dynamics
- Learning Smooth and Expressive Interatomic Potentials for Physical Property Prediction
- On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology
- The Llama 3 Herd of Models
- Scaling Laws for Autoregressive Generative Modeling
- Training Compute-Optimal Large Language Models
- Triplet Interaction Improves Graph Transformers: Accurate Molecular Graph Learning with Triplet Graph Transformers
- All-atom Diffusion Transformers: Unified generative modelling of molecules and materials
- In-Datacenter Performance Analysis of a Tensor Processing Unit
- Understanding and Mitigating Distribution Shifts For Machine Learning Force Fields
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- The Open Molecules 2025 (OMol25) Dataset, Evaluations, and Models
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks