Strong interplay of fundamental order parameters in complex oxides are known to give rise to exotic physical phenomena. The 4d transition metal oxide SrRhO3 has generated much interest, but advances have been hindered by difficulties in preparing single crystalline phases. Here, we have epitaxially stabilized high quality single crystalline SrRhO3 films and investigated their structural, electronic, and magnetic properties. We determine that their properties significantly differ from the paramagnetic metallic ground state that governs bulk samples and are strongly related to rotations of the RhO6 octahedra.