(347b) Sampling-Based Vs. Surrogate-Based Techniques for Data-Driven Optimization: A Comparative Study of Adaptive Sampling and Hybrid-Modeling Approaches

High-Fidelity (HF) simulations (including dynamic PDE systems, digital-twin models etc.) are mechanistic mathematical representations of physical systems that are often used for decision-making in chemical engineering [1, 2]. However, some of the challenges when optimizing with embedded HF simulations are: (a) the lack of objective function and constraint equations and derivatives, and (b) high-computational cost. To optimize such systems, there is a large body of literature dating back to 1980s [3] comprising of proposed algorithms that cleverly use “samples” without equations or derivatives to locate optima. With recent advances in Machine Learning (ML), surrogate-based techniques are frequently used, especially for cases the simulation is expensive to evaluate. However, surrogate-based methods require more effort to select and train a good surrogate model, and this creates uncertainty and slow convergence. Excellent efforts have been published recently to answer some of these open questions (i.e., which ML model? how to globally optimize ML models?)[4-7]. Overall, the vast majority of the surrogate-based optimization literature has focused on adaptive sampling and black-box ML models, while any explicitly known constraints are incorporated separately in grey-box formulations. Recently, there is a significantly increasing body of literature on physics-informed ML and/or hybrid modeling methods [8]. While the surrogate-based optimization and PIML literatures have been developing in parallel, we believe there are many unexplored connections and interesting comparisons between the two. In this study, we focus on integrating the recent advances in hybrid modeling and physics-informed ML, and provide new opportunities for improvement at the intersection of both the approaches.

Hybrid models (HM) combine multiple fidelity of models, to create more accurate composite models. For example, we have recently shown that a "Model Correction" hybrid architecture, that utilizes Neural Networks to relate high and low-fidelity data, leads to a composite model that learns from HF and low-fidelity data with improved accuracy [9]. In this work we first integrate HMs within surrogate-based techniques and quantify convergence, reliability, sampling, and CPU speed-ups due to hybridization. Moreover, we will show comprehensive comparison of the above strategy, versus increased a-priori sampling, state-of-the-art tuning of parameters and hyper-parameters of a single HM model, optimized by global solvers in full or reduced-space formulations. With the rise of tools like OMLT [10] that facilitate formulating ML models into python optimization environments such as Pyomo, this comparison is an interesting avenue for exploration. Finally, we compare hybrid structures built using reduced ordered modeling (ROM) methods for optimization [11]. Here, we look at various optimization approaches to computationally expensive models that allow for faster computation with minimal impact on the fidelity of the solution, by extending past work to employ Neural Network-based autoencoders and neural differential equation models for nonlinear dimensionality reduction.

To compare the computational costs of these approaches, we not only show results on global optimization benchmarks, but also focus on a complex chemical engineering case studies modeled by dynamic simulations, i.e., pressure swing adsorption for process design and gas separation applications. Ultimately, we will present a comprehensive comparison between different types of mathematical integration of mechanistic equations and ML, and traditional sampling or black-box surrogate-based optimization.

References:

1. Sokolov, M., et al., Hybrid modeling—a key enabler towards realizing digital twins in biopharma? Current Opinion in Chemical Engineering, 2021. 34: p. 100715-100715.
2. Sansana, J., et al., Recent trends on hybrid modeling for Industry 4.0. Computers & Chemical Engineering, 2021. 151: p. 107365-107365.
3. Jacobson, S. and L. Schruben, A review of techniques for simulation optimization. 1986.
4. Garud, S.S., N. Mariappan, and I.A. Karimi, Surrogate-based black-box optimisation via domain exploration and smart placement. Computers & Chemical Engineering, 2019. 130: p. 106567-106567.
5. Schweidtmann, A.M. and A. Mitsos, Deterministic global optimization with artificial neural networks embedded. Journal of Optimization Theory and Applications, 2019. 180(3): p. 925-948.
6. Williams, B., et al., PRESTO: Predictive REcommendation of Surrogate models To approximate and Optimize. Chemical Engineering Science, 2022. 249: p. 117360-117360.
7. Zhai, J. and F. Boukouvala, Data-driven spatial branch-and-bound algorithms for box-constrained simulation-based optimization. Journal of Global Optimization, 2022. 82(1): p. 21-50.
8. Bradley, W., et al., Perspectives on the integration between first-principles and data-driven modeling. Computers & Chemical Engineering, 2022: p. 107898-107898.
9. Ravutla, S. and F. Boukouvala, Hybrid Modeling and Multi-Fidelity Approaches for Data-Driven Branch-and-Bound Optimization. Computer Aided Chemical Engineering, 2023.
10. Ceccon, F., et al., OMLT: Optimization & machine learning toolkit. The Journal of Machine Learning Research, 2022. 23(1): p. 15829-15836.
11. Agarwal, A., L.T. Biegler, and S.E. Zitney, Simulation and optimization of pressure swing adsorption systems using reduced-order modeling. Industrial & Engineering Chemistry Research, 2009. 48(5): p. 2327-2343.