Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithm

This paper investigates whether deep architectures are essential for physics-informed neural networks (PINNs) by proposing the use of shallow neural networks combined with the Levenberg–Marquardt (LM) algorithm. By reformulating PINNs as nonlinear systems, the authors demonstrate that shallow architectures can solve both forward and inverse problems for nonlinear partial differential equations (PDEs) with high efficiency and accuracy, potentially reducing computational costs.

How it works

The researchers propose using shallow feed-forward neural networks with two hidden layers to approximate solutions. Instead of the standard approach of minimizing a composite loss function using first-order optimizers like Adam or quasi-Newton methods like LBFGS, they reformulate the training process as solving a nonlinear system of equations, or equivalently a nonlinear least-squares problem.

The Levenberg–Marquardt (LM) algorithm is employed to optimize network parameters, acting as a robust and efficient optimization method that blends the Gauss–Newton algorithm and gradient descent. To facilitate this, the authors develop an exact analytical differentiation scheme to compute the Jacobian matrix. This approach involves:

• Rewriting network operations in a layer-wise form for clarity.

• Deriving analytical expressions for derivatives with respect to input variables x and t.

• Constructing the Jacobian matrix J from the partial derivatives of each residual with respect to all network parameters.

Problem Formulations

For forward problems, the objective is to approximate the solution by minimizing the PDE residual. The authors enforce initial and boundary conditions directly within the network representation using a transformation that improves training efficiency and solution accuracy.

In inverse problems, unknown parameters lambda are identified simultaneously with the solution u. The Jacobian matrix for these problems takes a specific block form to handle both network weights and PDE parameters:

1. J 1: Derivatives of the PDE residuals with respect to the network weights.

2. J 2: Derivatives of the data-matching residuals with respect to the network weights.

3. J 3: Derivatives of the PDE residuals with respect to the unknown parameters lambda.

Experimental Results

The proposed method was tested on several benchmark problems to evaluate its performance. These included:

• The Burgers equation.

• The nonlinear Schrödinger equation.

• The Allen–Cahn equation.

• The three-dimensional Bratu equation.

Numerical results demonstrate that LM significantly outperforms BFGS in terms of convergence speed, accuracy, and final loss values. For the Burgers equation, LM achieved a final loss of 8.1 times 10-8, which is significantly smaller than the 1.4 times 10-3 obtained by BFGS. In the Schrödinger equation case, the shallow network attained a relative L2 error of 2.1 times 10-5, representing an improvement of nearly two orders of magnitude in accuracy while requiring more than 25 times fewer parameters compared to deeper models. For inverse problems, LM converges to the true parameter values both faster and more accurately than BFGS.

Implications of Network Depth

The study concludes that network depth and size alone do not determine the accuracy of PINNs when an appropriate optimization strategy is employed. Shallow architectures, when combined with effective second-order optimization methods, can provide accurate and computationally efficient solutions for both forward and inverse problems. This suggests that deep architectures may not be essential for high-fidelity results in a wide class of PDEs.

1. Implementation of Shallow-Architecture PINN Modules

• Improvement: Replace deep, high-parameter neural architectures with parsimonious, two-hidden-layer structures (NN(a, m 1, m 2, 1)) specifically optimized for physical constraints.

• Capability: The improved AI system can solve complex nonlinear partial differential equations (PDEs) using significantly fewer trainable parameters (up to 25x reduction) while maintaining or exceeding the accuracy of deep models. This enables high-fidelity physical simulations on edge devices and low-resource hardware with minimal memory footprints.

2. Integration of Levenberg–Marquardt (LM) Optimization Engines

• Improvement: Shift from first-order (Adam) or quasi-Newton (LBFGS) optimizers to a second-order Levenberg–Marquardt algorithm specifically formulated for nonlinear least-squares residual minimization.

• Capability: The AI system can achieve much higher precision, reaching loss values in the 10-7 to 10-10 range, where standard optimizers often stagnate at 10-3. This results in faster convergence and more stable training for highly nonlinear systems like the Schrödinger or Burgers equations.

3. Transition from Automatic Differentiation to Exact Analytical Derivative Schemes

• Improvement: Incorporate the paper’s derived analytical expressions for spatial, temporal, and weight-based derivatives into the backpropagation pipeline, replacing reliance on standard automatic differentiation (AD).

• Capability: The system can perform extremely rapid and computationally efficient Jacobian matrix evaluations. This reduces the computational overhead per iteration and provides a transparent, mathematically rigorous dependency structure between network weights and the resulting differential operators, enhancing training stability.

4. Unified Forward-Inverse Nonlinear System Solver

• Improvement: Reformulate PINN training from simple loss minimization to solving a unified nonlinear system F(W) = 0.

• Capability: The AI system can perform simultaneous state estimation (forward problem) and high-precision parameter identification (inverse problem). For example, it can identify unknown physical constants (e.g., diffusion coefficients or reaction rates) in real-time with extremely low absolute percentage errors (APE), making it ideal for autonomous system calibration and digital twin synchronization.

This work investigates shallow physics-informed neural networks (PINNs) for solving forward and inverse problems governed by nonlinear partial differential equations (PDEs). By formulating PINN training as a nonlinear least-squares problem, the Levenberg-Marquardt (LM) algorithm is used to efficiently optimize the network parameters. Exact analytical expressions for neural-network derivatives with respect to the input variables are derived, revealing the relationships between the network output and its spatial and temporal derivatives and providing a clearer interpretation of the PINN architecture. These expressions are then used to derive explicit formulas for the Jacobian matrix required by LM. The proposed approach is evaluated on the Burgers, Schrödinger, Allen-Cahn, and three-dimensional Bratu equations. Numerical results show that LM substantially outperforms BFGS, L-BFGS, and Adam in convergence speed, accuracy, and final loss values. Comparisons with deeper networks further demonstrate that shallow LM-PINNs can achieve higher accuracy with substantially fewer parameters, emphasizing the importance of considering network architecture and optimization strategy jointly. The explicit analytical Jacobian also provides computational and memory advantages that are particularly relevant to large-scale PINNs. Overall, these results suggest that, for a broad class of PDEs, shallow PINNs combined with effective second-order optimization can provide accurate and computationally efficient solutions to both forward and inverse problems.