(345i) A Comparative Analysis on Interpretability of Explainable AI (XAI) for Neural Network Based Fault Detection Methods

Artificial intelligence (AI) technology has spread to many fields, achieving astounding performance in certain cases. A single most important factor responsible for this is the development of deep learning using multiple hidden layers. Despite the popularity, deep neural networks (DNN) are black box models and have important limitations, among which first and foremost is that lack of physical interpretability. Explainable AI (XAI), which attempts to explain how the AI reached its decisions, has been getting a lot of attention lately in expectation of drawing physical insights from the black box models.

Recent developments in the field of XAI enable humans to comprehend the decision making of AI, by analyzing the contribution of input features. For example, an explanation can be represented as a heatmap highlighting which pixels of the input image are most relevant to the classification decision [1-3], or that highlighting texts in sentences [4, 5]. A large number of XAI methods have been put forward in the domain of image classification and natural language processing. Surveys providing comparative overview of these techniques also have appeared [6, 7].

In the field of process system engineering, many efforts have been reported to apply AI technologies to process modeling, control and optimization. Fault detection and isolation (FDI) is one of the areas where AI technologies have been popular, with various detection and classification models ranging from feedforward neural networks to SVM and LSTM [8-10]. As FDI’s goal is to ensure safety and on-spec product quality, the lack of interpretability of these approaches hinder their widespread applications. Motivated by this, this work applies representative XAI methods to DNN models for fault detection and compares their performance in identifying the most important input features for the detection.

The XAI methods we examine are Integrated gradients, DeepLIFT, Kernel SHAP, Gradient SHAP. Integrated gradients computes the average gradient along the path from a given baseline to the input [11]. DeepLIFT is based on the backpropagation approach, which attributes a change to inputs based on the differences between the inputs and the baselines [12]. Kernel SHAP uses a specially-weighted local linear regression to estimate the Shapley values [13]. Gradient SHAP is a gradient method to compute the Shapley values [13]. For comparison, DNN are trained to perform fault detection for the Tennessee Eastman process according to [14] and each XAI method is applied on the models respectively to compute each input feature’s attribution to detecting the fault. Once most relevant input variables are chosen, neural networks are then trained again without those variables. We compare the interpretation ability of the various methods by computing the difference in the fault detection rate between the original and newly trained models and analyzing the difference. As a result, among the four methods tried, Gradient SHAP showed the largest difference in the fault detection rate. As proper interpretability enhances the acceptance of an AI model for fault detection, effective application of the XAI methods is expected to accelerate the adoption of AI in the process engineering field.

References:

[1] Simonyan, K., A. Vedaldi, and A. Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps. 2013.

[2] Landecker, W. and M.D. Thomure..., Interpreting individual classifications of hierarchical networks. 2013.

[3] Bach, S., et al., On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. 2015.

[4] Li, J., et al., Visualizing and understanding neural models in nlp. 2015.

[5] Arras, L., et al., " What is relevant in a text document?": An interpretable machine learning approach. 2017.

[6] Alvarez, D., On the robustness of interpretability methods. 2018.

[7] Danilevsky, M., et al., A survey of the state of explainable AI for natural language processing. 2020.

[8] Yin, S., et al., Study on support vector machine-based fault detection in tennessee eastman process. 2014.

[9] Gao, X. and J. Hou, An improved SVM integrated GS-PCA fault diagnosis approach of Tennessee Eastman process. 2016.

[10] Zhao, H., S. Sun, and B. Jin, Sequential fault diagnosis based on LSTM neural network. 2018.

[11] Sundararajan, M., A. Taly, and Q. Yan, Axiomatic attribution for deep networks. 2017.

[12] Shrikumar, A. and P. Greenside..., Learning important features through propagating activation differences. 2017.

[13] Lundberg, S. and S.I. Lee, A unified approach to interpreting model predictions. 2017.

[14] Heo, S. and J.H. Lee, Fault detection and classification using artificial neural networks. 2018.