(235g) Computer Vision Aided Process Control: Methods for Enhanced Autonomy and Robustness

Computer vision strategies enable the automatic extraction of desired information from visual data. This powerful capability has proved valuable in diverse fields, such as: autonomous vehicle environment sensing [1], facial recognition [2], automatic inspection [3], biomedical imaging analysis [4], and quality control [5]. This has inspired emergent application in process control where Convolutional Neural Network (CNN) sensors map visual data to a state signal that can be interpreted by the controller [6]. Such sensors enable the use of complex visual data (which traditionally has been interpreted manually by operators) to be incorporated into automatic control loops. Hence, this technology allows us to reap the benefits of automatic control (e.g., increased profitability, safety, and efficiency) for processes that previously had to be controlled manually by operators [6].

We rigorously introduce this new control paradigm and highlight its advantages and challenges. One key challenge we address is how incorporating such sensors introduces a significant system operation vulnerability because CNN sensors can exhibit high prediction errors when exposed to novel (abnormal) visual data. Unfortunately, identifying such novelties in real-time is nontrivial. We review existing novelty detection approaches such as reconstruction models and one-class classification (OCC) which provide rich capabilities for novelty detection with general datasets [7]; however, in our context their principal disadvantage is that their latent space is independent of the feature space used by the CNN sensor, potentially making them overly conservative in identifying novelties that may have little effect on the CNN’s performance.

To address this issue, we propose the Sensor-Activated Feature Extraction One-Class Classification (SAFE-OCC) framework which leverages the convolutional blocks of the CNN to create an effective feature space to conduct novelty detection [8]. This approach engenders a feature space that directly corresponds to that used by the CNN sensor and avoids the need to derive an independent latent space (which typically requires creating/training complex deep learning models). These advantages make the SAFE-OCC approach for assessing the quality of predictions made by a CNN sensor in real-time as required for process control applications. We demonstrate the effectiveness of SAFE-OCC via simulated control environments.

References:

[1] Janai, Joel, Fatma Güney, Aseem Behl, and Andreas Geiger. "Computer vision for autonomous vehicles: Problems, datasets and state of the art." Foundations and Trends® in Computer Graphics and Vision 12, no. 1–3 (2020): 1-308.

[2] Balaban, Stephen. "Deep learning and face recognition: the state of the art." Biometric and surveillance technology for human and activity identification XII 9457 (2015): 68-75.

[3] Neethu, N. J., and B. K. Anoop. "Role of computer vision in automatic inspection systems." International Journal of Computer Applications 123, no. 13 (2015).

[4] Suzuki, Kenji. "Overview of deep learning in medical imaging." Radiological physics and technology 10, no. 3 (2017): 257-273.

[5] Wu, Di, and Da-Wen Sun. "Colour measurements by computer vision for food quality control–A review." Trends in Food Science & Technology 29, no. 1 (2013): 5-20.

[6] Martynenko, Alex. "Computer vision for real-time control in drying." Food Engineering Reviews 9, no. 2 (2017): 91-111.

[7] Ruff, Lukas, Jacob R. Kauffmann, Robert A. Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G. Dietterich, and Klaus-Robert Müller. "A unifying review of deep and shallow anomaly detection." Proceedings of the IEEE (2021).

[8] Pulsipher, Joshua L., Luke DJ Coutinho, Tyler A. Soderstrom, and Victor M. Zavala. "SAFE-OCC: A Novelty Detection Framework for Convolutional Neural Network Sensors and its Application in Process Control." arXiv preprint arXiv:2202.01816 (2022).