This paper analytically and numerically investigates misalignment and mode-mismatch induced power coupling coefficients and losses as a function of Hermite Gauss (HG) mode order. We show that higher-order HG modes are more susceptible to beam perturbations when, for example, coupling into optical cavities: the misalignment and mode-mismatch-induced power coupling losses scale linearly and quadratically with respect to the mode indices respectively. As a result, the mode-mismatch tolerance for the $mathrm{HG}_{3,3}$ mode is reduced to a factor of 0.28 relative to the currently-used $mathrm{HG}_{0,0}$ mode. This is a potential hurdle to using higher-order modes to reduce thermal noise in future gravitational-wave detectors.