We report on measurements and modeling of the mode structure of tunable Fabry-Perot optical microcavities with imperfect mirrors. We find that non-spherical mirror shape and finite mirror size lead to loss, mode deformation, and shifted resonance fre
quencies at particular mirror separations. For small mirror diameters, the useful cavity length is limited to values significantly below the expected stability range. We explain the observations by resonant coupling between different transverse modes of the cavity and mode-dependent diffraction loss. A model based on resonant state expansion that takes into account the measured mirror profile can reproduce the measurements and identify the parameter regime where detrimental effects of mode mixing are avoided.
