2020 Virtual AIChE Annual Meeting

(280g) Multi-Stage Chemical Plant Production Scheduling Under Uncertainty with Deep Reinforcement Learning

Authors

Hubbs, C. D. - Presenter, The Dow Chemical Company
Sahinidis, N. - Presenter, Carnegie Mellon University
Grossmann, I., Carnegie Mellon University
Wassick, J., The Dow Chemical Company
Multi-Stage Chemical Plant Production Scheduling under

Uncertainty with Deep Reinforcement Learning

Christian Hubbs1,2, Nikolaos Sahinidis1, Ignacio Grossmann1, and John Wassick2

1Department of Chemical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213

2Digital Fulfillment Center, Dow Chemical, Midland, MI 48642

Keywords: Machine Learning, Reinforcement Learning, Optimization, Scheduling

1 Abstract

Industrial chemical operations require complex questions regarding resource allocation to be asked and answered continuously in order to meet operational and profitability targets [Harjunkoski et al., 2014]. Employees who work as planners and schedulers take decisions on production, storage vs. selling, timing, and sourcing all under significant uncertainty: the plant may experience an interruption due to maintenance; customers may decide to change their order, either the timing of the delivery or the product; interruptions upstream may cause raw materials to be delayed; pricing may fluctuate, impacting the margins and thus trade-offs between these decisions. These changes may transform a previously optimal schedule into a sub- optimal or infeasible schedule.

Optimization under uncertainty has received significant attention by the process systems engineering community and a plethora of approaches have been developed. While explicitly accounting for uncertainty can greatly improve the result of the model above deterministic formulations, this improved result comes with a trade-off [Huang and Ahmed, 2009]. Models including uncertainty lead to significantly higher computa- tional costs due to a large number of scenarios in cases where discrete uncertainty is present. Computational costs of models which represent uncertainty as continuous probability distributions are driven by integration requirements (Balasubramanian and Grossmann, 2003, Sahinidis, 2004).

To address these issues, we apply machine learning techniques, namely deep reinforcement learning, to scheduling a multi-stage, continuous production process. We build upon the work presented in Hubbs et al. [2020] by adding a second, packaging stage, finer time intervals, and interactions with other sites in the network. This serves to make the model more general and widely applicable to large-scale industrial processes. Moreover, uncertainty from production delays and maintenance outages are introduced into the model, as are customer priority levels to more accurately capture the relevant trade-offs faced by the motivating, industrial case.

The RL agent is trained using Monte Carlo simulation over thousands of simulated years in order to learn a policy that maximizes the profitability of the system. These additions greatly expand the scope and complexity of the planning and scheduling decisions, closely mirroring the real-world decision process. While training must take place off-line using a simulation, once trained, the model can then be deployed on-line in a production system to yield real-time schedules as new orders are entered into the enterprise resource planning (ERP) system and as the situation in the plant changes due to delays and unplanned events.

References

J. Balasubramanian and I. Grossmann. Scheduling optimization under uncertainty—an alternative approach.

Computers and Chemical Engineering, 27:469–490, 2003. ISSN 0098-1354. doi: 10.1016/S0098-1354(02)

00221-1.

D. Bertsimas, D. B. Brown, and C. Caramanis. Theory and Applications of Robust Optimization. SIAM Review, 53(3):464–501, 2011. ISSN 0036-1445. doi: 10.1137/080734510. URL http://epubs.siam.org/ doi/10.1137/080734510.

I. E. Grossmann, R. M. Apap, B. A. Calfa, P. Garc´ıa-Herreros, and Q. Zhang. Recent advances in math- ematical programming techniques for the optimization of process systems under uncertainty. Computers and Chemical Engineering, 91:3–14, 2016. ISSN 00981354. doi: 10.1016/j.compchemeng.2016.03.002. URL http://dx.doi.org/10.1016/j.compchemeng.2016.03.002.

I. Harjunkoski, C. T. Maravelias, P. Bongers, P. M. Castro, S. Engell, I. E. Grossmann, J. Hooker, C. M´endez, G. Sand, and J. Wassick. Scope for industrial applications of production scheduling models and solution methods. Computers and Chemical Engineering, 62:161–193, 2014. ISSN 00981354. doi: 10.1016/j. compchemeng.2013.12.001. URL http://dx.doi.org/10.1016/j.compchemeng.2013.12.001.

K. Huang and S. Ahmed. The Value of Multistage Stochastic Programming in Capacity Planning Under

Uncertainty. Operations Research, 57(4):893–904, 2009. ISSN 0030-364X. doi: 10.1287/opre.1080.0623.

C. D. Hubbs, C. Li, N. V. Sahinidis, I. E. Grossmann, and J. M. Wassick. A Deep Reinforcement Learning

Approach for Chemical Production Scheduling. Forthcoming, 2020.

X. Lin, S. L. Janak, and C. A. Floudas. A new robust optimization approach for scheduling under uncertainty: I. Bounded uncertainty. Computers and Chemical Engineering, 28(6-7):1069–1085, 2004. ISSN 00981354. doi: 10.1016/j.compchemeng.2003.09.020.

N. V. Sahinidis. Optimization under uncertainty: State-of-the-art and opportunities. In Computers and

Chemical Engineering, volume 28, pages 971–983, 2004. ISBN 0098-1354. doi: 10.1016/j.compchemeng.

2003.09.017.