(372y) Invertible Neural Networks As Discrete Time Process Models for Correlated Non-Gaussian Noise

Discrete-time models (DTMs) are well-established process models for dynamical systems and often provide
the basis for modern control schemes such as model predictive control (MPC) (Rawlings et al., 2017). In
their standard form, DTMs predict the system states at the time step x(t + 1) as a function of the current
states x(t), and the control inputs u. Typically, DTMs are estimated from process data and, thus, are some
of the earliest adoptions of machine learning in chemical engineering (Söderström, 2002) and, recently,
machine learning models such as neural networks have become popular choices of DTMs (Schweidtmann
et al., 2021). Industrial processes are subject to noise and uncertainty. Thus, DTMs are often written as
stochastic models by including noise terms in the prediction (Söderström, 2002). However, simple noise
terms are limited to Gaussian (white) noise and fail to describe possible correlations between the system
states. For correlated uncertainties and non-Gaussian noise, such simple descriptions fail to represent
the real system stochastics accurately. This work uses invertible neural networks (INNs) (Papamakarios
et al., 2021) to design probabilistic DTMs. Previous works by the author have used INNS as a forecasting
method for renewable electricity generation (Cramer et al., 2022a, 2023). However, the concept generalizes
to any other type of probabilistic regression task. INNs are inherently multivariate and can be designed as
bijective transformations of multivariate Gaussians to form explicit probability distribution models called
normalizing flows (Papamakarios et al., 2021). If the current states x(t) are combined with the controls u
in the conditional inputs, normalizing flows can be used as autoregressive models, i.e., probabilistic DTMs
that predict a conditional probability distribution p(x(t + 1)|x(t), u) (Cramer et al., 2022b). The INN is
used to model the dynamics of the Van-der-Vusse reactor (Kravaris and Daoutidis, 1990) subject to high
noise levels. The INN accurately learns the process behavior, and the predicted mean values follow the
actual state value. Using INNs as DTMs allows users to describe complex high-dimensional systems with
correlated, skewed, or even multimodal probability distributions. This work is a first step towards many
applications, including probabilistic MPC and other uses of chance-constrained programming.

References:

- Cramer, E., Mitsos, A., Tempone, R., and Dahmen, M. (2022a). Principal component density estimation
for scenario generation using normalizing flows. Data-Centric Engineering, 3:e7.

- Cramer, E., Paeleke, L., Mitsos, A., and Dahmen, M. (2022b). Normalizing flow-based day-ahead wind
power scenario generation for profitable and reliable delivery commitments by wind farm operators.
Computers & Chemical Engineering, 166:107923.

- Cramer, E., Witthaut, D., Mitsos, A., and Dahmen, M. (2023). Multivariate probabilistic forecasting of
intraday electricity prices using normalizing flows. Applied Energy, 346:121370.

- Kravaris, C. and Daoutidis, P. (1990). Nonlinear state feedback control of second-order nonminimum-
phase nonlinear systems. Computers & Chemical Engineering, 14(4-5):439–449.

- Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2021).
Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research,
22(57):1–64.

- Rawlings, J. B., Mayne, D. Q., Diehl, M., et al. (2017). Model predictive control: theory, computation,
and design, volume 2. Nob Hill Publishing Madison, WI.

- Schweidtmann, A. M., Esche, E., Fischer, A., Kloft, M., Repke, J.-U., Sager, S., and Mitsos, A. (2021).

- Machine learning in chemical engineering: A perspective. Chemie Ingenieur Technik, 93(12):2029–2039.
Söderström, T. (2002). Discrete-time Stochastic Systems: Estimation and Control. Springer London.