Lipid nanoparticles (LNPs) have emerged as effective carriers for mRNA delivery in vaccine and therapeutic applications, attracting substantial attention since the COVID-19 pandemic. Continued efforts are crucial to optimize LNP composition for improved delivery efficacy and to elucidate the underlying mechanisms driving differences in protein expression. This study systematically screened PEGylated lipids for intramuscular mRNA delivery, followed by optimization of the formulation composition, physicochemical characterization, and investigation of the structure–activity relationship (SAR). Using a model ionizable lipid, we initially evaluated a library of PEGylated lipids, each varying in linker chemistries, tail structures, or PEG molecular weights. 1,2-Dimyristoyl-rac-glycero-3-methoxypolyethylene glycol – 5000 (DMG-PEG5k) was identified as a promising candidate from this screening. Using a design of experiments (DoE) approach, we further optimized the formulation to increase in vivo transfection efficacy. To explore the SAR of the DoE formulations, advanced physicochemical characterization was conducted including Laurdan assay, SAXS, Cryo-TEM, and QCM-D, alongside standard LNP analysis. Among the key factors examined, high mRNA encapsulation efficiency, LNP membrane integrity (especially under acidic conditions), and ordered internal structures were identified as the critical parameters for transfection efficiency. Overall, balancing the ratios of all LNP components is critical for maximizing LNP functionality. This study presents a systematic evaluation and characterization of LNPs with different PEG-lipid moieties, deepens SAR understanding, and provides valuable guidelines for rationally designing more effective next-generation LNPs.