Reliable mechanistic operator recovery with biologically-informed neural networks: principles for architecture and optimisation design

Reliable mechanistic operator recovery with biologically-informed neural networks (BINNs) addresses the challenge of inferring unknown biological mechanisms directly from sparse and noisy experimental data by embedding governing differential equations into neural network training. The gist: successful mechanistic inference is achieved by appropriately balancing model expressivity, optimisation, physical consistency, and data informativeness rather than maximising any single aspect of the model or optimisation.

The BINN Framework

Biologically-informed neural networks (BINNs) combine neural networks with mechanistic differential equations to enable the recovery of unknown biological processes from sparse datasets. The framework is designed to infer interpretable constitutive operators directly from observational data by representing each operator—diffusion, advection, and reaction—with a separate neural network. The overall BINN model consists of four Multi-Layer Perceptrons (MLPs): one for the latent solution, one for the diffusivity function, one for the advection velocity function, and one for the reaction term. These networks are trained jointly using automatic differentiation to enforce the governing equation as a soft constraint during optimisation.

Benchmark Models and Data Generation

The study systematically investigates these principles across a suite of canonical one-dimensional advection–diffusion–reaction partial differential equation (PDE) models. These models span increasing levels of mechanistic complexity, including linear diffusion, nonlinear diffusion, reaction–diffusion, and nonlinear advection. The synthetic datasets used for training are generated by solving these PDEs using known constitutive functions and parameters from Table 3. To mimic biological measurement uncertainty and intrinsic variability, synthetic observations are created by perturbing the exact or numerical solutions with independent and identically distributed (i.i.d.) Gaussian noise, denoted as uo(xi, ti) = u(xi, ti) + N (0, σ2).

Key Factors Governing Reliable Inference

The paper systematically investigates how several design choices influence optimisation behaviour and operator recovery:

1. Network expressivity: The study shows that Moderately expressive architectures outperform overly complex networks. Increasing network capacity beyond a moderate level rarely improves reconstruction accuracy once the underlying constitutive relationships are represented, as further increases provide little additional benefit and may lead to over-fitting.

2. Learning rate: The learning rate is crucial for stability; intermediate learning rates balance efficient exploration of the loss landscape with optimisation stability. Very small rates lead to under-fitting, while excessively large rates cause unstable optimisation or even divergence.

3. Loss weighting: Accurate operator recovery requires an appropriate balance between data-fitting and PDE residual losses. The total loss function is defined as Ltotal(θ, ϕ, ζ, ψ) = wdataLdata(θ) + wPDELPDE(θ, ϕ, ζ, ψ) + wBCLBC(θ), where the weights (wdata ≥ 0, wPDE ≥ 0, and wBC ≥ 0) mediate this trade-off.

4. Batch size: The batch size determines the stochasticity of optimisation. Intermediate batch sizes provide the best compromise between gradient stochasticity, computational efficiency, and reproducibility. Small batches encourage exploration but increase computational cost; very large batches reduce stochasticity, leading to slower progress per unit time.

Diagnostics and Practical Recommendations

The research identifies practical diagnostics for recognising common failure modes when ground truth is unavailable:

(Over-fitting)

(Unstable optimisation)

(Poor mechanistic recovery)

The findings establish evidence-based guidelines, suggesting that reliable mechanistic inference is achieved not by maximising network complexity or optimisation effort, but by appropriately balancing model expressivity, optimisation, physical consistency, and data informativeness. Practitioners are advised to monitor quantities such as the relative errors between the learned solution (uθ) and the forward-predicted solution (up), as agreement between up and observed dynamics provides stronger evidence that the underlying mechanisms have been correctly identified. Furthermore, it is recommended that Data-fitting should remain at least as influential as the PDE residual throughout optimisation, either through iterative training strategies or by ensuring that wdata ≳ wPDE. Intermediate learning rates and batch sizes are consistently identified as providing the most favourable compromise.

Robustness to Noise and Data Informative

The performance of BINNs is assessed across varying degrees of noise and data density:

(Observation Noise)

(Data Informativeness)

BINNs are robust to moderate observation noise, but increasing variance leads to greater variability between independent initialisations. Increasing the quantity of data consistently improves both solution and operator recovery, but this improvement is limited by the range of solution values represented in the data. The study concludes that BINN performance depends more strongly on the informativeness of available data than on simply increasing model complexity.

(Data Sampling Density)

The results for densely sampled datasets show that increasing the quantity of data improves both solution and operator recovery, but this does not alter the qualitative conclusions regarding optimal architectural or optimisation choices.

Here are specific improvements to AI systems based on the findings of this research, along with what these improved systems could achieve:

The core takeaway is that reliable mechanistic inference in Biologically-Informed Neural Networks (BINNs) is achieved not by maximizing complexity, but by balancing four competing objectives: model expressivity, optimization strategy, physical consistency (PDE residuals), and data informativeness.

Here are the specific improvements and resulting capabilities:

1. Implementation of Multi-Objective Loss Weighting Strategies

2. Adaptive Learning Rate Scheduling based on Validation Loss Trajectories

3. Architecture Selection Guided by Data Informativeness Metrics

4. Dynamic Batch Size Adjustment for Robust Optimization

Specific Improvements and Capabilities:

1. The AI system should move beyond a single, fixed total loss function (Eq. 19) and implement a mechanism to dynamically adjust the weights of the data-fitting loss, PDE residual loss, and boundary condition loss based on real-time training progress.

2. The system should incorporate an adaptive learning rate scheduler that monitors validation loss curves (as seen in Fig. 3) to automatically adjust the step size during optimization—increasing it when convergence is slow and decreasing it sharply if unstable or erratic trajectories are detected, thus avoiding overshooting good minima or divergence.

3. The architecture design module should incorporate a metric for data informativeness (e.g., analyzing the range of solution values covered by training data, as suggested in Section 4) to guide the selection of network depth and width for the constitutive operator networks (Dϕ, Vζ, Gψ). The system should avoid over-parameterization when the data range is narrow or noisy.

4. The optimization engine should employ a dynamic batch size strategy that transitions between small batches (to encourage exploration of complex loss landscapes) and intermediate batches (to maintain computational efficiency and reproducibility), optimizing for the lowest relative reconstruction error across both solution networks and operator networks, as demonstrated in Fig. 6.

What the Improved AI System Can Do:

The resulting improved BINN system will be significantly more reliable for discovering unknown biological mechanisms from sparse, noisy experimental data:

1. Robust Mechanistic Operator Recovery (The Primary Goal)

2. Distinguishing Interpolation from Mechanism Discovery

3. Handling Data Sparsity and Noise Effectively

Specifically, the improved AI system can:

1. Identify the true underlying physical mechanisms (e.g., density-dependent diffusion coefficients in tissue growth or nonlinear advection velocities in cell migration) with a much higher degree of confidence than current static BINN models.

2. Accurately predict future system states by simulating the governing biological dynamics using the recovered, physically consistent operators, even when experimental data is sparse and noisy (e.g., predicting tumor invasion patterns based on limited biopsy data).

3. Provide reliable diagnostic feedback during training: Instead of just reporting a final loss value, the system will flag specific failure modes (over-fitting, unstable optimization) in real-time, allowing researchers to intervene before significant computational resources are wasted on unreliable models.

Many biological processes are governed by complex dynamical mechanisms that remain incompletely understood despite increasing volumes of experimental data. Biologically-informed neural networks (BINNs) seek to address this challenge by embedding differential equations into neural network training, enabling constitutive operators to be recovered directly from sparse and noisy observations. However, the extent to which operator recovery depends on architectural design, optimisation strategy and the information within the data is not yet well understood. We present an empirical study of how these factors influence mechanistic inference using BINNs applied to one-dimensional advection-diffusion-reaction partial differential equations. Across a suite of problems, we investigate how network expressivity, learning rate, loss weighting and batch size influence optimisation behaviour, reconstruction accuracy and operator recovery. We show that mechanistic inference is governed by balancing competing objectives rather than maximising any single aspect. Moderately expressive architectures outperform complex networks, intermediate learning rates balance efficient exploration with optimisation stability, accurate operator recovery requires a balance between data-fitting and PDE residual losses and intermediate batch sizes provide the best compromise between efficient parameter space exploration, computational efficiency and reproducibility. We further identify practical diagnostics for recognising common failure modes, including over-fitting, unstable optimisation and poor mechanistic recovery. These findings establish guidelines for deploying BINNs as credible tools for biological model discovery and demonstrate that reliable mechanistic inference is achieved by appropriately balancing model expressivity, optimisation, physical consistency and data informativeness.