(4ix) Probabilistic Regression Using Conditional Invertible Neural Networks: Usecases in Forecasting and Process Modeling in Energy- and Chemical Engineering

Predicting process behavior is one of the central tasks in process systems engineering. If there is insuffi-
cient knowledge about the process to build mechanistic models, machine learning models, and particularly
regression models, are often the tool of choice. Regression models such as neural networks have proven to
be functional tools for discrete-time modeling of chemical processes and forecasting important parameters
(Gonzaga et al., 2009). However, many processes exhibit stochastic behavior that cannot be represented
using standard regression models. Instead, process models, i.e., regression models, need to represent the
inherent uncertainty of the process (Mesbah et al., 2022), e.g., via probabilistic regression. Normalizing
flows use invertible neural networks (INNs) to project high dimensional data onto a multivariate Gaus-
sian (Papamakarios et al., 2021). The inverse of the INN then maps the Gaussian to the data space, which
enables sampling and generation of new data. Normalizing flows make no assumptions about the modeled
probability distributions and, thus, provide a highly flexible tool to describe and sample high-dimensional
real-world data. Furthermore, clever designs of the INNs allow for explicit computation of the probabil-
ity density function (PDF) of arbitrary datasets via the change of variables formula (Dinh et al., 2015,
2017). The training of normalizing flows uses this explicit PDF to train via log-likelihood maximiza-
tion, leading to stable and consistent convergence (Rossi, 2018). If normalizing flows are complemented
with conditional information, i.e., external inputs to the INN, the resulting conditional normalizing flow
presents a probabilistic regression model for high-dimensional data (Cramer et al., 2022; Rasul et al.,
2021). The conditional input may comprise any combination of numerical values, e.g., relevant regres-
sors, encoded labels, or autoregressive features. Thus, conditional normalizing flows build highly flexible,
nonlinear, probabilistic regression models for high dimensional data with complicated probability distri-
butions. This poster presents applications of normalizing flow-based probabilistic regression for energy-
and chemical engineering applications. In particular, the poster highlights the normalizing flow’s per-
formance in scenario generation of renewable power generation time series with subsequent stochastic
optimization (Cramer et al., 2022), probabilistic multi-period forecasting of day-ahead and intraday elec-
tricity prices (Hilger et al., 2024; Cramer et al., 2023), and probabilistic modeling of stochastic chemical
processes. The normalizing flow consistently outperforms alternative methods. For instance, the nor-
malizing flow-generated scenarios lead to more profitable decisions in stochastic optimization (Cramer
et al., 2022), and the probabilistic electricity price forecasts show much narrower yet accurate prediction
intervals (Cramer et al., 2023). Furthermore, the normalizing flow accurately learns the behavior of a
stochastic CSTR process. In summary, normalizing flows and their extension to conditional probability
distributions present a highly flexible problem-agnostic probabilistic regression tool for high-dimensional
data that consistently produces high-quality results. Future applications may include modeling stochastic
processes in biochemical engineering, modeling for stochastic optimal control, and generative modeling
for design.

Cramer, E., Paeleke, L., Mitsos, A., and Dahmen, M. (2022). Normalizing flow-based day-ahead wind
power scenario generation for profitable and reliable delivery commitments by wind farm operators.
Computers & Chemical Engineering, 166:107923.
Cramer, E., Witthaut, D., Mitsos, A., and Dahmen, M. (2023). Multivariate probabilistic forecasting of
intraday electricity prices using normalizing flows. Applied Energy, 346:121370.
Dinh, L., Krueger, D., and Bengio, Y. (2015). NICE: Non-linear independent components estimation. In
Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR
2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings.
Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017). Density estimation using Real NVP. In 5th In-
ternational Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017,
Conference Track Proceedings. OpenReview.net.
Gonzaga, J., Meleiro, L., Kiang, C., and Maciel Filho, R. (2009). Ann-based soft-sensor for real-time
process monitoring and control of an industrial polymerization process. Computers & Chemical Engi-
neering, 33(1):43–49.
Hilger, H., Witthaut, D., Dahmen, M., Rydin Gorjão, L., Trebbien, J., and Cramer, E. (2024). Mul-
tivariate scenario generation of day-ahead electricity prices using normalizing flows. Applied Energy,
367:123241.
Mesbah, A., Wabersich, K. P., Schoellig, A. P., Zeilinger, M. N., Lucia, S., Badgwell, T. A., and Paulson,
J. A. (2022). Fusion of machine learning and mpc under uncertainty: What advances are on the
horizon? In 2022 American Control Conference (ACC). IEEE.
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2021).
Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research,
22(57):1–64.
Rasul, K., Sheikh, A.-S., Schuster, I., Bergmann, U. M., and Vollgraf, R. (2021). Multivariate probabilistic
time series forecasting via conditioned normalizing flows. In International Conference on Learning
Representations, Vienna, Austria.
Rossi, R. J. (2018). Mathematical statistics: An introduction to likelihood based inference. John Wiley &
Sons.