(169cs) Advancing Molecular Screening: Integrating Operating Conditions into Transformer Models for Chemical Engineering

The chemical industry has undergone considerable advancements, yet a pressing requirement for further development of processes to reduce energy consumption, increase yield, and meet production demands persists. Engineers have multiple strategies at their disposal to boost the efficiency of processes. Among these, process optimization through intensification[1], heat integration[2], and the advancement of chemicals used as solvents, catalysts, and promoters are particularly significant. The discovery of new chemicals is the most direct route to substantial enhancements within these approaches. Nonetheless, the experimental discovery of new solvents proves to be an exceedingly demanding task.

An alternative to experimental approaches is utilizing computational methods such as density functional theory and molecular dynamics. These techniques considerably alleviate the arduous process of conducting experiments. However, they are often associated with high computational costs and lengthy time requirements. Various machine learning-based methodologies have been adopted to surmount this challenge, owing to their reduced computational demands and rapid prediction capabilities[3][4]. These methods excel in making system-specific predictions and have advanced the development of rapid quantitative structure-property relationships. Nonetheless, traditional machine-learning approaches are limited in generalizing across different chemical systems. They frequently necessitate extensive feature engineering and are typically tailored for a limited range of compounds, restricting their applicability to other molecules. In contrast, recent progress has been made with transformer-based machine learning models. These models have demonstrated potential in cheminformatics, providing accurate predictions of system-specific properties using the SMILES notation of compounds. This represents a notable enhancement over previous models, primarily due to the self-attention mechanism, which facilitates a more profound understanding of intermolecular interactions. However, despite the significant advances in employing transformers to predict properties under standard conditions effectively, their application in forecasting properties under diverse conditions, essential for the design and optimization of industrial processes, remains an area in need of further exploration and development.

In our research, we have incorporated various operating conditions characteristic of chemical processes to enhance the molecular screening process, thereby aiding engineers in developing optimized processes. We employed a transformer model pre-trained on an extensive dataset comprising 1.6 billion molecules from the PubChem database. This pre-training enables the model to capture the complex relationships between chemical structures and their properties. The encoder within the transformer generates a numeric representation of molecules, which can be regarded as a 'universal chemical language' for property prediction. During the fine-tuning stage, the molecules are processed through the encoder to obtain molecular embeddings. These embeddings are subsequently combined with the up-projected operating conditions and input into a neural network with residual connections to predict the target property. Incorporating residual connections allows the neural network to retain the molecular correlations obtained from the encoder while giving equal consideration to the operating conditions introduced later. This approach addresses two significant challenges in utilizing transformers for property predictions in chemical engineering. Firstly, we obviate the need for a large labeled dataset to train the encoder by excluding operating conditions from the pre-training phase. This is particularly beneficial given the scarcity of high-quality data in many chemical engineering domains, which has previously impeded the application of transformers in these areas. Secondly, we augment the model's relevance for process design by incorporating operating condition augmentations. Given the dynamic nature of chemical processes, the ability to predict properties under varying conditions is essential for designing more efficient processes. This methodology enables its application in a broad spectrum of chemical systems for purposes including, but not limited to, catalyst discovery, drug development, and solvent exploration from a process design perspective. Our architecture minimizes the computational burden of training transformers for different systems and ensures accurate predictions without compromising efficiency. To validate the framework's effectiveness, we conducted a case study on screening amine-based solvents for carbon capture. This case study demonstrated the framework's efficiency in predicting properties under varying conditions and enabling engineers to perform solvent-screening tasks specific to the nature of the carbon capture process.

References:
[1] Sophie Sitter, Qi Chen, and Ignacio E. Grossmann. "An Overview of Process Intensification Methods." Current Opinion in Chemical Engineering, vol. 25, 2019, pp. 87-94.
[2] Guzman-Urbina, Alexander, et al. "Systematic Process Energy Optimization via Multi-level Heat Integration: A Case Study on Low-Temperature Reforming for Methanol Synthesis." Computer Aided Chemical Engineering, edited by Yoshiyuki Yamashita and Manabu Kano, vol. 49, Elsevier, 2022, pp. 1207-1212.
[3] Chen, Guangying, et al. "Artificial Neural Network Models for the Prediction of CO2 Solubility in Aqueous Amine Solutions." International Journal of Greenhouse Gas Control, vol. 39, 2015, pp. 174-184.
[4] Haghighatlari, Mojtaba, et al. "Learning to Make Chemical Predictions: The Interplay of Feature Representation, Data, and Machine Learning Methods." Chem, vol. 6, no. 7, 2020, pp.