The Fe
3O
4 nanoparticles been successfully functionalized by (3-Mercaptopropyl) trimethoxy-silane (Fe
3O
4-SH) and Vinyl ester resin (VER)/Fe
3O
4 with and without thiol groupnanocomposites have been fabricated at different loadings of Fe
3O
4 nanoparticles. Fourier transform infrared spectroscopy (FT-IR) is used to characterize the chemical structureof Fe
3O
4-SH. X-ray diffraction (XRD) reveals that the crystallization peak of Fe
3O
4-SH nanoparticles have moved to the small angle slightly. The grafting ratio of thiol group and thermal stability of the VER/Fe
3O
4nanocompositesare were characterized by thermogravimetric analysis (TGA). Mechanical properties are performed by tensile test (TS) and dynamic mechanical analysis (DMA). Compared to pure non-magnetic VER composites, VER/Fe
3O
4 with and without thiol-group nanocomposites have good magnetization with the loading of Fe
3O
4