Tetragonal Mn$_x$Ga$_{1-x}$ (x=0.70, 0.75) thin films grown on SrTiO$_3$ substrates at different temperatures and thicknesses exhibit perpendicular magnetic anisotropy with coercive fields between 1-2 T. Transmission electron microscopy (TEM) and X-ray diffraction (XRD) reveal that 40nm samples grown at 300-350$^{o}$C lead to polycrystalline films with the tetragonal c-axis oriented primarily perpendicular to the film plane but with some fraction of the sample exhibiting the c-axis in the film plane. This structure results in a secondary magnetic component in the out of plane magnetization. Growth at 300$^{o}$C with a reduced thickness or Mn concentration significantly decreases the presence of the tetragonal c-axis in the film plane, thus improving the magnetic properties. TEM is of critical importance in characterizing these materials, since conventional XRD cannot always identify the presence of additional crystallographic orientations although they can still affect the magnetic properties. Our study points to ways that the microstructure of these thin films can be controlled, which is critical for utilization of this material in spintronic devices.