Intermediate mass ratio inspiral (IMRI) binaries -- containing stellar-mass black holes coalescing into intermediate-mass black holes ($M>100M_{odot}$) -- are a highly anticipated source of gravitational waves (GWs) for Advanced LIGO/Virgo. Their detection and source characterization would provide a unique probe of strong-field gravity and stellar evolution. Due to the asymmetric component masses and the large primary, these systems generically excite subdominant modes while reducing the importance of the dominant quadrupole mode. Including higher order harmonics can also result in a $10%-25%$ increase in signal-to-noise ratio for IMRIs, which may help to detect these systems. We show that by including subdominant GW modes into the analysis we can achieve a precise characterization of IMRI source properties. For example, we find that the source properties for IMRIs can be measured to within $2%-15%$ accuracy at a fiducial signal-to-noise ratio of 25 if subdominant modes are included. When subdominant modes are neglected, the accuracy degrades to $9%-44%$ and significant biases are seen in chirp mass, mass ratio, primary spin and luminosity distances. We further demonstrate that including subdominant modes in the waveform model can enable an informative measurement of both individual spin components and improve the source localization by a factor of $sim$10. We discuss some important astrophysical implications of high-precision source characterization enabled by subdominant modes such as constraining the mass gap and probing formation channels.