(392bk) Hyperparameter Tuning of Physics-Informed Machine Learning Using Multi-Objective Optimization

Physics-informed neural networks (PINNs) have emerged as a rapidly advancing paradigm within the intersection of machine learning and engineering systems [1]. By embedding physical laws into the loss function of neural networks, PINNs can guide model training toward physically consistent solutions rather than relying purely on data-driven approaches. In contrast to traditional machine learning methods, which might require a large amount of labeled data or risk violating physical constraints, PINNs can better handle scenarios involving limited data, complex boundary conditions, or noisy measurements. This capability has attracted significant interest and enabled a wide range of applications in fields such as fluid dynamics, material science, chemical engineering, and more.

A typical PINN framework incorporates multiple loss terms to constrain the solution space, including the PDE residual loss, initial condition loss, and boundary condition loss [2]. These different loss terms collectively guide the network to approximate the true solution while satisfying physical constraints. To balance the individual contributions of each term to the overall gradient, one usually introduces scaling factors. Manual adjustment of these weights through grid or random search can be both time-consuming and prone to error, especially when the number of loss terms grows [3]. Although several studies have explored methods to adapt the relative importance of different loss terms, these approaches often remain non-adaptive across different training stages and can therefore miss the ideal balance as the network develops. The difficulty is further exacerbated under parameter uncertainty, such as in chemical engineering processes, where the first-principles model parameters might be partially unknown. In these cases, fixed or heuristically selected weights may lead to suboptimal solutions, necessitating a more flexible and adaptive framework to balance and adjust for loss terms.

From a broader optimization perspective, the training of a PINN can be reformulated as a multi-objective optimization (MOO) problem, where each loss term represents a unique objective to be minimized. MOO techniques offer a more comprehensive exploration of the solution space and provide valuable Pareto frontier information, which highlights the trade-offs among competing objectives and offers multiple potential solutions [4]. However, at this stage, adaptively adjusting the weights of each term through multi-objective techniques has not been investigated.

Motivated by the above considerations, in this talk, we will introduce a hyperparameter tuning method for physics-informed machine learning using multi-objective optimization. Specifically, we treat different loss terms as separate objectives in a multi-objective setting. The non-dominated sorting genetic algorithm II (NSGA-II) is employed to jointly optimize these objectives, discovering a Pareto frontier of solutions that reflect different trade-offs among the multiple loss terms. By leveraging the rich information provided by Pareto-optimal solutions, we explore the competition and synergy between different objectives and further assign appropriate weights to each loss term. In this way, the network can adaptively schedule the importance of each loss term during training, avoiding the limitations and deviations caused by manually setting weights. Through a number of experiments on representative chemical engineering case studies, we demonstrate the effectiveness of the proposed framework in escaping local minimum and accelerating convergence, especially in the face of parameter uncertainty.

References:

[1] Wu, G., Yion, W. T. G., Dang, K. L. N. Q., & Wu, Z. (2023). Physics-informed machine learning for MPC: Application to a batch crystallization process. Chemical Engineering Research and Design, 192, 556-569.

[2] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378, 686-707.

[3] Wang, S., Teng, Y., & Perdikaris, P. (2021). Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5), A3055-A3081.

[4] Wang, Z., Parhi, S. S., Rangaiah, G. P., & Jana, A. K. (2020). Analysis of weighting and selection methods for pareto-optimal solutions of multiobjective optimization in chemical engineering applications. Industrial & Engineering Chemistry Research, 59(33), 14850-14867.