(286a) Fusion of EEG and Eye-Tracking Based Metrics for Characterizing the Cognitive State of Control Room Operators

Process industries are highly hazardous, and the continued occurrence of accidents points toward a need to develop more robust and resilient safety barriers. Several layers of protection prevent hazards from transforming into accidents, and humans form one of the most critical layers. The importance of humans as a layer of protection can be envisioned by the fact that most of the accidents in process industries trace their roots to human failures. Human errors account for over 60-80% of accidents in process industries (Brauer, 2016). An analysis of severe industrial accidents from 2008 to 2018 reveals that more than 76.1% can be attributed to human failures (Jung et al., 2020). With an increased digitalization and concomitant rise in automation in process industries, the role of humans has evolved. The human roles primarily involve monitoring during routine operations and diagnosis and prognosis during abnormalities. These roles have become cognitively challenging due to a need to acquire, segregate, and integrate information from multiple sources to maintain optimal operating conditions. A requirement to process an overabundance of information increases distraction and cognitive workload— precursors to suboptimal performance— if the operators’ capabilities do not match process demands (Porthin et al., 2021). This paper addresses the critical need to understand and best evaluate cognitive aspects of control room operators’ performance using a fusion of eye-tracking and electroencephalography (EEG)-based metrics.

Traditional methods of human performance assessment ignore cognitive aspects of operators’ performance. On the contrary, the performance is assessed primarily based on metrics such as the number of successes/failures, deviations of operator actions from standard operating procedures, and response time (Lee et al., 2000). Such metrics fail to account for the underlying reasons (e.g., cognitive workload, shortcomings in mental models) and contributory factors (e.g., the effect of HMI design) to sub-optimal performance. Advancements in cognitive engineering and concurrent development of sensor technologies such as eye-tracking and electroencephalography (EEG) have made it possible to obtain several physiological measures that provide critical insights into human cognition and performance. Eye-tracking primarily provides information about how an operator acquires information from HMIs (reflection of operator’s mental models) (Shahab et al., 2022) and how HMI designs influence performance (Diego-Mas et al., 2019). Likewise, electroencephalography (EEG) provides critical information about operators’ mental states, such as levels of cognitive workload (Liang et al., 2018). Nonetheless, eye-tracking alone provides limited insight into the mental processes; brain-sensing can complement these processes. For instance, a classification accuracy of 93% for visual hazard detection was attained when features from eye-tracking and EEG were fused compared to accuracies of 73% (for eye-tracking) and 83% (for EEG) obtained independently (Noghabaei et al., 2021). Our previous research using eye-tracking (Sharma et al., 2016, Bhavsar et al., 2017) and EEG (Iqbal et al., 2020, 2021) reveals exciting insights into the cognitive behavior of control room operators. For instance, experts have more directed attention as captured by lower gaze entropy than novices (Bhavsar et al., 2017). EEG is able to identify instances when there is a mismatch between the operator’s mental models of process and actual process behavior (Iqbal et al., 2020).

In the current work, we study the added benefit of the fusion of eye-tracking and EEG for operator performance assessment. The proposed approach for fusion involves identifying key events during a typical period during which an operator interacts with the process. These key events could include: the periods during which an operator is simply monitoring the process when it runs normally; periods which involve disturbances; and periods during which an operator is actively taking control actions to bring the process back within normal operating conditions. The purpose of the approach is to identify which metrics—eye-tracking and/or EEG-based—can best capture operator’s cognition during these events, and provide insights into their cognitive performance. For instance, whether eye-tracking based metrics are more sensitive to performance (say, level of attention) during monitoring or EEG-based. Likewise, which metrics can quickly capture increase in cognitive workload in the wake of inability of an operator to control process abnormalities, and which respond after a lag. To obtain answers to these questions, we use a simulated ethanol production process as an experimental testbed for conducting human subject studies. Ten participants were involved in the study, and each of them carried out several repetitive trials. During a typical trial, a participant has to carry out six tasks (each corresponding to a disturbance scenario). Overall, these participants performed 81 trials resulting in 438 task-participant pairs. During these tasks, participants' brain activity and eye gaze are recorded via a single electrode EEG sensor and Tobii 300 eye tracker, respectively. In addition, process data, alarm information, and operator actions are also recorded. These are used to create labels such as the monitoring phase, the period immediately after an alarm, before an operator's action, after an operator's first slider action, and the period during which the operator executes a control action. Next, eye-tracking and EEG-based features are extracted for each of these labels. These features include average pupil diameter, power spectral densities in the range of 0 to 2 Hz for pupil diameter, and power spectral densities in the range of 0-30 Hz for EEG. Decision tree-based analysis of features reveals that combining features from eye and EEG data provides more insights into operator cognitive performance than either of these. This is hypothesized due to differences in dynamics (e.g., timescale) of EEG and eye data. In this paper, we will present the proposed fusion methodology and present results from our human subject studies that demonstrate the benefits of the fusion of eye tracking and EEG based metrics.

Acknowledgement: This work is partially funded by American Express Lab for Data Analytics,

Risk and Technology, Indian Institute of Technology Madras

References

Bhavsar, P., Srinivasan, B., & Srinivasan, R. (2017). Quantifying situation awareness of control room operators using eye-gaze behavior. Computers & chemical engineering, 106, 191-201.

Brauer, R. L. (2016). Safety and health for engineers. John Wiley & Sons.

Diego-Mas, J. A., Garzon-Leal, D., Poveda-Bautista, R., & Alcaide-Marzal, J. (2019). User-interfaces layout optimization using eye-tracking, mouse movements and genetic algorithms. Applied ergonomics, 78, 197-209.

Iqbal, M. U., Shahab, M. A., Choudhary, M., Srinivasan, B., & Srinivasan, R. (2021). Electroencephalography (EEG) based cognitive measures for evaluating the effectiveness of operator training. Process Safety and Environmental Protection, 150, 51-67.

Iqbal, M. U., Srinivasan, B., & Srinivasan, R. (2020). Dynamic assessment of control room operator’s cognitive workload using Electroencephalography (EEG). Computers & Chemical Engineering, 141, 106726.

Jung, S., Woo, J., & Kang, C. (2020). Analysis of severe industrial accidents caused by hazardous chemicals in South Korea from January 2008 to June 2018. Saf. Sci. 124, 104580

Lee, S., Jeong, I., & Il, M. (2000). Development of evaluation algorithms for operator training system. Computers & Chemical Engineering, 24(2-7), 1517-1522.

Liang, Y., Liang, W., Qu, J., & Yang, J. (2018, October). Experimental study on EEG with different cognitive load. In 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 4351-4356). IEEE.

Noghabaei, M., Han, K., & Albert, A. (2021). Feasibility Study to Identify Brain Activity and Eye-Tracking Features for Assessing Hazard Recognition Using Consumer-Grade Wearables in an Immersive Virtual Environment. Journal of Construction Engineering and Management, 147(9), 04021104.

Porthin, M., Liinasuo, M., & Kling, T. (2020). Effects of digitalization of nuclear power plant control rooms on human reliability analysis–A review. Reliability Engineering & System Safety, 194, 106415.

Shahab, M. A., Iqbal, M. U., Srinivasan, B., & Srinivasan, R. (2022). HMM-based models of control room operator’s cognition during process abnormalities. 1. formalism and model identification. Journal of Loss Prevention in Process Industries, 104748.

Sharma, C., Bhavsar, P., Srinivasan, B., & Srinivasan, R. (2016). Eye gaze movement studies of control room operators: A novel approach to improve process safety. Computers & Chemical Engineering, 85, 43-57.