(644c) A Simultaneous Approach for Training Neural Differential-Algebraic Systems of Equations

Scientific machine learning is an emerging field that broadly describes the combination of scientific computing and machine learning to address challenges in science and engineering. Within the context of differential equations, this has produced highly influential methods, such as physics-informed neural networks (PINNs) [1], neural ordinary differential equations (NODEs) [2] and universal differential equations (UDEs) [3]. Recent works extend this line of research to consider neural differential-algebraic systems of equations (DAEs) [4,5,6], where some unknown relationships within the DAE are learned from data. Sequential approaches for training neural DAEs, similarly to neural ODEs, are computationally expensive, as they require the solution of a DAE for every parameter update. Further, the rigorous consideration of algebraic constraints is difficult within common deep learning training algorithms such as stochastic gradient descent.

In this work [7], we apply the simultaneous approach for dynamic optimization to the problem of training neural DAEs. This results in a fully discretized nonlinear optimization problem, where the weights of the neural network as well as the differential-algebraic states are optimization variables. We extend recent work applying a pseudo-spectral approach to neural ODEs [8], by presenting a general framework to solve neural DAEs, with explicit consideration of hybrid models, where some components of the DAE are known, e.g. physics-informed constraints. We focus on computational strategies to improve the tractability of the nonlinear optimization problem resulting from the simultaneous approach. This includes a specialized initialization scheme based on the solution of an auxiliary, tractable optimization problem. Furthermore, we propose the use of Hessian approximations for constraints related to the neural components of the DAE within the interior point method (IPM) used to solve the nonlinear optimization problem. We show that this helps alleviate computational bottlenecks arising from the dense and non-convex nature of neural networks. Lastly, we present a generic implementation of our approach using the Pyomo modeling framework [9], where the evaluation of neural components is handled through external functions relying on popular deep learning libraries to efficiently compute Jacobians and Hessians. We demonstrate promising results in terms of accuracy, model generalizability and computational cost on estimation tasks from a variety of DAE systems. We believe that this work provides promising computational evidence that the simultaneous approach for DAE-constrained optimization has the potential to enhance the landscape of computational tools used within the growing field of scientific machine learning, and provides the opportunity to expand the practicality of nonconvex nonlinear programming methods to the domain of hybrid model training.

References

[1] M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019. ISSN 0021-9991. doi:https://doi.org/10.1016/j.jcp.2018.10.045.

[2] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

[3] Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385, 2020

[4] Tannan Xiao, Ying Chen, Shaowei Huang, Tirui He, and Huizhe Guan. Feasibility study of neural ode and dae modules for power system dynamic component modeling. IEEE Transactions on Power Systems, 38(3):2666–2678, 2022.

[5] Christian Moya and Guang Lin. Dae-pinn: a physics-informed neural network model for simulating differential algebraic equations with application to power networks. Neural Computing and Applications, 35(5):3789–3804, 2023

[6] Vincenzo Di Vito, Mostafa Mohammadian, Kyri Baker, and Ferdinando Fioretto. Learning to solve differential equation constrained optimization problems. arXiv preprint arXiv:2410.01786, 2024.

[7] Laurens R. Lueg, Victor Alves, Daniel Schicksnus, John R. Kitchin, Carl D. Laird and Lorenz T. Biegler. A Simultaneous Approach for Training Neural Differential-Algebraic Systems of Equations. arxiv preprint arXiv:2504.04665, 2025

[8] Mariia Shapovalova and Calvin Tsay. Training neural odes using fully discretized simultaneous optimization. arXiv preprint arXiv:2502.15642, 2025

[9] Michael L. Bynum, Gabriel A. Hackebeil, William E. Hart, Carl D. Laird, Bethany L. Nicholson, John D. Siirola, Jean-Paul Watson, and David L. Woodruff. Pyomo–optimization modeling in python, volume 67. Springer Science & Business Media, third edition, 2021.