(588bq) Multi-Constrained Molecular Optimization in Transformer-Derived Latent Spaces

Designing molecules that satisfy multiple simultaneous constraints—such as efficacy, solubility, permeability, metabolic stability, and safety—is a fundamental challenge in early-stage drug and material discovery. While generative models like GANs, VAEs, and reinforcement learning-based frameworks have made strides in exploring chemical space, they typically operate by sampling from a learned distribution. As a result, they struggle to solve true multi-objective optimization problems. These models are not inherently designed to enforce multiple property constraints explicitly, often requiring complex and unstable loss formulations or multi-step post-processing. This leads to issues such as poor control over generated properties, limited generalizability to new constraints, and a high risk of generating impractical molecules [1].

To address the limitations of sampling-based generative models, we present a new framework that approaches molecular design as a direct optimization task in the latent space of a transformer-based chemical language model. Rather than generating molecules and filtering them post hoc, our method searches over a continuous, chemically meaningful embedding space where multiple property constraints can be enforced simultaneously and explicitly. This latent space is defined by a transformer encoder pre-trained on large-scale SMILES datasets using masked language modeling [2,3], which produces rich molecular representations that capture both local structural features and broader functional context—without relying on handcrafted descriptors. Optimization is carried out in this latent space using Particle Swarm Optimization (PSO), a metaheuristic algorithm well-suited for navigating high-dimensional, non-convex search spaces [4]. In this setting, each particle represents a candidate molecule in latent space, and its quality is assessed through a suite of neural network-based property predictors. These predictors are trained to estimate key attributes such as lipophilicity, solubility, permeability, metabolic clearance, and toxicity, directly from the latent embeddings. By combining all property requirements into a single composite objective function, the framework guides the search toward embeddings that correspond to chemically valid molecules satisfying multiple target criteria. This structured and constraint-aware optimization process enables more reliable and targeted molecular discovery compared to conventional generative approaches.

Once optimal latent vectors are identified, we use a greedy decoding mechanism to reconstruct molecular SMILES from the embeddings, ensuring that generated structures are syntactically valid and chemically meaningful. Unlike traditional generative models, which often require iterative tuning or fine-tuning to meet multiple objectives, our approach enables direct and interpretable optimization in latent space, improving efficiency, scalability, and constraint adherence. A key forward-looking aspect of our work is its integration potential with process modeling. While current efforts focus on optimizing molecular-level properties, many real-world applications demand alignment with process-level metrics—such as ease of formulation, solvent recovery, or separation efficiency. Our framework is designed to interface with simulation tools that model such downstream behavior. By passing predicted molecular properties into process models, we enable system-level evaluation and selection of generated molecules based not only on intrinsic merit but also on real-world functional compatibility.

In summary, we introduce a scalable and modular methodology for multi-constrained molecular optimization that overcomes key limitations of existing generative approaches. By combining transformer-based molecular embeddings, metaheuristic optimization, and property-aware decoding—with extensibility toward process modeling—this framework lays the groundwork for more informed, controlled, and application-ready molecular design.

References:

[1] D. Saxena, J. Cao, Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions, ACM Comput. Surv. 54 (2021). https://doi.org/10.1145/3446374.

[2] A. Khambhawala, C.H. Lee, S. Pahari, P. Nancarrow, N.A. Jabbar, M.M. El-Halwagi, J.S.-I. Kwon, Advanced transformer models for structure-property relationship predictions of ionic liquid melting points, Chemical Engineering Journal 503 (2025) 158578. https://doi.org/https://doi.org/10.1016/j.cej.2024.158578.

[3] A. Khambhawala, C.H. Lee, S. Pahari, J.S.-I. Kwon, Minimizing late-stage failure in drug development with transformer models: Enhancing drug screening and pharmacokinetic predictions, Chemical Engineering Journal (2025) 160423. https://doi.org/https://doi.org/10.1016/j.cej.2025.160423.

[4] J. Kennedy, R. Eberhart, Particle swarm optimization, in: Proceedings of ICNN’95 - International Conference on Neural Networks, 1995: pp. 1942–1948 vol.4. https://doi.org/10.1109/ICNN.1995.488968.