SrMoO$_3$ is a promising material for its excellent electrical conductivity, but growing high-quality thin films remains a challenge. Here we synthesized epitaxial films of SrMoO$_3$ using the molecular beam epitaxy (MBE) technique under a low oxygen-flow rate. Introduction of SrTiO$_3$ buffer layers of 4--8 unit cells between the film and the (001)-oriented SrTiO$_3$ or KTaO$_3$ substrate was crucial to remove impurities and/or roughness of the film surface. The obtained film shows improved electrical conductivities as compared with films obtained by other techniques. The high quality of the SrMoO$_3$ film is also verified by angle-resolved photoemission spectroscopy (ARPES) measurements showing clear Fermi surfaces.