(711g) Causality Guided Minimization of Liquid Time Constant Neural Network Models

Liquid time constant (LTC) models, inspired by the neural circuitry found in nature, have shown significant promise for modeling complex dynamical systems, such as chemical industrial processes, as state observer models recovering causal pathways between input features and outputs via training and backpropagation. These models are particularly useful in scenarios where the underlying physical relationships are not well understood or are too complex to be captured by traditional modeling techniques. LTC models can adapt to the dynamics of the system, making them suitable for a wide range of applications in process control, fault detection, and optimization.

However, the utility of LTC models can be limited by large input dimensionality, which leads to several challenges. Firstly, high-dimensional input spaces can hinder the interpretability of the model, making it difficult to understand the relationships between input features and the model's predictions. Secondly, large input dimensionality can result in increased computational complexity, both during training and inference, which can be a significant bottleneck in real-time applications. Additionally, the cost and maintenance of sensors required to capture high-dimensional input data can be a practical limitation in industrial settings.

To address these challenges, we propose an algorithm that systematically minimizes LTC models by leveraging the causal structure inherent in the LTC framework. The key idea is to identify and retain only the most informative input features while maintaining the model's predictive accuracy. The proposed approach consists of two main steps: intervention coefficient analysis and iterative feature exclusion. In the first step, we perform intervention coefficient analysis to quantify the impact of each input feature on the model's states. This is achieved by calculating a node sensitivity matrix for a fully trained LTC model. The node sensitivity matrix captures the sensitivity of each model state to perturbations in the input features. By analyzing the magnitude of the intervention coefficients, we can identify the input features that have the most significant influence on the model's behavior. In the second step, we iteratively exclude lower impact features and retrain the observer model until the desired balance between feature minimization and accuracy is reached. The exclusion of less informative features reduces the input dimensionality, leading to a more compact and interpretable model. The retraining process allows the model to adapt to the reduced input space and maintain its predictive performance.

To demonstrate the effectiveness of the proposed approach, we apply it to synthetic data simulated from three different traditionally modeled system types in mechanical, chemical, and ecological engineering. These systems represent a diverse range of dynamical behaviors and provide a comprehensive evaluation of the causality-guided minimization algorithm. The results show that the minimized models identified by the causality-guided approach converge faster during training compared to the original high-dimensional models. This faster convergence can be attributed to the reduced input dimensionality, which simplifies the optimization landscape and allows for more efficient parameter updates during backpropagation. Moreover, the minimized models achieve better test accuracy across most error metrics compared to models trained on all available input features. This improved accuracy suggests that the causality-guided approach effectively identifies the most informative input features and eliminates redundant or noisy information that can hinder the model's generalization performance.

The proposed causality-guided minimization algorithm offers several benefits for the application of LTC models in real-world scenarios. By reducing the input dimensionality, it enhances the interpretability of the models, allowing domain experts to gain insights into the underlying causal relationships. The reduced computational complexity enables faster training and inference times, making the models more suitable for real-time monitoring and control applications. Furthermore, the minimized models require fewer sensors, reducing the associated costs and maintenance requirements.

In conclusion, we present a novel approach for minimizing liquid time constant neural network models using a causality-guided algorithm. By leveraging the causal structure of the LTC framework and performing intervention coefficient analysis, we systematically identify and exclude less informative input features. The minimized models demonstrate improved convergence speed and test accuracy across different engineering domains. This work contributes to the development of more efficient and interpretable LTC models, enhancing their applicability in real-world industrial settings. Future research directions include extending the approach to handle time-varying systems, incorporating domain knowledge into the feature selection process, and exploring the scalability of the algorithm to larger-scale industrial processes.