(204c) Learning Control Invariant Set By Leveraging Default Value Function in Rl

Computing the control invariant set (CIS) of a general nonlinear system is critical for ensuring safe operation under state and control constraints. However, it is challenging due to nonlinearity and dimensionality. Existing methods, such as those relying on set-based or grid-based techniques [1, 2] and those tailored to polynomial representations [3], often encounter scalability issues or impose restrictive assumptions that are
overly conservative in practice.

In this work, we propose a reinforcement learning (RL) based approach for computing the CIS of a class of nonlinear systems. Specifically, we formulate the CIS problem as an RL task with a tailored reward design and zero-initialized value functions. Under this formulation, the value associated with states that are within the CIS remains at zero, whereas states that are not within the CIS will have a negative value. In tabular RL,
the CIS can then be identified directly by collecting all grid points whose learned values are zero, whereas in deep RL, a threshold-based criterion is used to separate the CIS from other states. We also show that the proposed reward and initialization scheme guarantee consistent convergence of safe states to zero values in the absence of disturbances. Once trained, the RL provides a simple characterization of the CIS in the form such that the value function equals zero or greater than the threshold.

We illustrate the effectiveness of the proposed RL-based method through a continuously stirred tank reactor (CSTR) subject to operating constraints on states (concentration, and temperature), and on manipulated input (coolant flow rate). In this example, our RL formulation yields CIS approximations that align well with graph-based solutions [4]. The present work offers a scalable and conceptually straightforward tool for CIS computation and at the same time offers a simple characterization of the obtained CIS, which holds great potential for applications.

[1] Tyler Homer and Prashant Mhaskar. Constrained control Lyapunov function-based control of nonlinear systems. Systems & Control Letters, 110:55–61, 2017.

[2] Ian M Mitchell, Alexandre M Bayen, and Claire J Tomlin. A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on automatic control, 50(7):947–957, 2005.

[3] Milan Korda, Didier Henrion, and Colin N Jones. Convex computation of the maximum controlled invariant set for polynomial control systems. SIAM Journal on Control and Optimization, 52(5):2944–2969, 2014.

[4] Benjamin Decardi-Nelson and Jinfeng Liu. Computing robust control invariant sets of constrained nonlinear systems: A graph algorithm approach. Computers & Chemical Engineering, 145:107177, 2021.