(394ah) A Human-Centric Explanation of Reinforcement Learning Agents Using Large Language Models

Recent advancements in reinforcement learning (RL) have driven significant progress in control and optimization domains, attributed to its model-free nature and ability to handle nonlinearity and uncertainty. The emergence of deep reinforcement learning (DRL) has further enhanced agent performance in high-dimensional spaces by employing deep neural networks (DNN) as function approximators [1-3]. Despite these advancements, the black-box nature of DNNs often renders the agent’s decision-making process obscure and difficult to trust, which is particularly critical in safety-sensitive applications such as chemical process control [4,5].

While several explainable RL (XRL) approaches have been proposed to address this issue, they are not yet fully comprehendible to domain experts due to persistent challenges. First, the distinct goal of RL from other machine learning as well as its dynamic nature has led to a wide range of user queries and corresponding XRL methods, but users don’t know which XRL methods to use. Also, most existing XRL approaches lack interactivity, limiting users’ ability to engage in follow-up exploration. Lastly, the absence of domain-specific knowledge in generated explanations often requires additional interpretation by human experts.

To address these challenges, we present TalkToAgent, a natural language-based, universal interactive chatbot designed to explain RL agent behaviors to end users. Built on GPT-4 [6], TalkToAgent first maps diverse XRL queries raised by control experts to corresponding explanation strategies, each implemented as API calls. After retrieving XRL results from the relevant APIs, the system integrates them with domain knowledge, which is incorporated into the explainer model through prompt engineering. After the the results are summarized into natural language form, TalkToAgent is prompted to propose relevant follow-up questions, facilitating additional understanding through interactive dialogue.

We demonstrate the effectiveness of TalkToAgent through its application to regularization and maximization tasks in controlling the chemical reactor [7]. User studies indicate that TalkToAgent delivers informative and comprehendible explanations, even for non-expert end users, thereby bridging the gap between complex RL methods and human interpretability.

[1] Daoutidis, Prodromos, et al. "Machine learning in process systems engineering: Challenges and opportunities." Computers & Chemical Engineering 181 (2024): 108523.

[2] Zhu, Xinji, Yujia Wang, and Zhe Wu. "Reinforcement learning for optimal control of stochastic nonlinear systems." AIChE Journal (2025): e18840.

[3] Kim, Yeonsoo, and Jong Woo Kim. "Safe model‐based reinforcement learning for nonlinear optimal control with state and input constraints." AIChE Journal 68.5 (2022): e17601.

[4] Milani, Stephanie, et al. "Explainable reinforcement learning: A survey and comparative review." ACM Computing Surveys 56.7 (2024): 1-36.

[5] Szatmári, Kinga, et al. "Resilience-based explainable reinforcement learning in chemical process safety." Computers & Chemical Engineering 191 (2024): 108849.

[6] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., ... & McGrew, B. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

[7] Park, Joonsoo, et al. "Reinforcement Learning for Process Control: Review and Benchmark Problems." International Journal of Control, Automation and Systems 23.1 (2025): 1-40.