(74c) Surrogate Quality Score: A Hybrid Metric to Balance Model Accuracy and Complexity

With improvements in data collection and storage capabilities, continuous evolution of computer processors, and rapid advances in the domain of artificial intelligence, data-driven-based machine learning models (surrogate models) are finding increased usage in modeling industrial processes and systems. Given the availability of numerous ML modeling techniques and associated model configurations, selecting the ideal surrogate form entails a few important considerations. These include interrogating data characteristics such as nonlinearity, dynamics, multicollinearity between predictors, response noise, etc. and the use case of the trained models (such as global metamodeling, data-driven optimization), etc. These considerations influence one to define a pool of candidate surrogate forms from which one or multiple forms can be potentially selected. Often, the selected surrogate form from such a pool is based on modeling familiarity and system-specific knowledge. In the absence of substantial domain knowledge or modeling expertise, one either needs to perform an exhaustive search over many candidate forms or use surrogate selection and building frameworks (Ahmad and Karimi, 2022, 2021; Cozad et al., 2014; Garud et al., 2018; Sun and Braatz, 2021; Williams and Cremaschi, 2019) to obtain the best surrogate forms. The common task involved in all these approaches is to balance the bias-variance tradeoff to obtain an optimal surrogate form that neither significantly underfits (large bias) nor overfits (large variance) on the observed data. To quantify the quality of trained surrogates based on this tradeoff, two approaches are commonly followed. The first approach involves computing the prediction errors on a validation data set, i.e. data set not used for training, using metrics such as mean absolute error, mean squared or root mean squared error, predictive coefficient of determination or predictive , etc. The second approach is useful when limited data exists, as often is the case in manufacturing processes. In such cases, it is important to use all available data to train the surrogate to capture maximal system information and estimate a surrogate’s generalizability or prediction capabilities by penalizing the prediction errors on the train set with model complexity. Metrics such as adjusted , Akaike Information Criterion (AIC), Minimum Description Length (MDL), Bayesian Information Criterion (BIC), etc. follow this approach to balance the bias error and variance.

With this discussion, the key question in modeling industrial processes and systems is whether a surrogate’s predictive accuracy should be the sole criterion to choose the best surrogate form? Often, process experts are interested in having accurate, yet simple and/or interpretable models. A simple surrogate might be necessary for surrogate-based optimization for maintaining tractability, or an interpretable model might be needed to correlate model predictions with theoretical evidence. Thus, the above question can be reframed as follows: Does the improvement in surrogate accuracy justify an increase in model complexity?

To this end, we formulate a hybrid comparative metric, Surrogate Quality Score (SQS), that balances surrogate accuracy with its complexity to compare and rank surrogates with varying accuracies and complexities. Our metric is developed heuristically, validated on several scenarios with differing model accuracies and complexities. SQS uses normalized root mean squared error on a validation set to interpret surrogate accuracy and degrees of freedom to quantify model complexity. Unlike metrics that penalize goodness-of-fit with a fixed complexity penalty (such as AIC, BIC, etc.), SQS has a tunable parameter to vary the balance between accuracy and complexity factors, thereby allowing the users to observe the best surrogates for varying degrees of accuracy-complexity tradeoffs. This could enable modeling experts to make informed decisions in selecting the ideal surrogate form with appropriate accuracy and simplicity based on the modeling objective. While SQS has been incorporated in our surrogate selection framework, LEAPS2 (Ahmad and Karimi, 2021), our modified SQS is more generic, robust, and versatile than the existing version in LEAPS2.

References:

Ahmad, M., Karimi, I.A., 2022. Families of similar surrogate forms based on predictive accuracy and model complexity. Comput. Chem. Eng. 163, 107845. https://doi.org/10.1016/j.compchemeng.2022.107845

Ahmad, M., Karimi, I.A., 2021. Revised learning based evolutionary assistive paradigm for surrogate selection (LEAPS2v2). Comput. Chem. Eng. 152, 107385. https://doi.org/10.1016/j.compchemeng.2021.107385

Cozad, A., Sahinidis, N.V., Miller, D.C., 2014. Learning surrogate models for simulation‐based optimization. AIChE J. 60, 2211–2227. https://doi.org/10.1002/aic.14418

Garud, S.S., Karimi, I.A., Kraft, M., 2018. LEAPS2: Learning based Evolutionary Assistive Paradigm for Surrogate Selection. Comput. Chem. Eng. 119, 352–370. https://doi.org/10.1016/j.compchemeng.2018.09.008

Sun, W., Braatz, R.D., 2021. Smart process analytics for predictive modeling. Comput. Chem. Eng. 144, 107134. https://doi.org/10.1016/j.compchemeng.2020.107134

Williams, B.A., Cremaschi, S., 2019. Surrogate Model Selection for Design Space Approximation And Surrogatebased Optimization, in: Computer Aided Chemical Engineering. Elsevier, pp. 353–358. https://doi.org/10.1016/B978-0-12-818597-1.50056-4