The double Penning trap mass spectrometer JYFLTRAP has been employed to measure masses and excitation energies for $11/2^-$ isomers in $^{121}$Cd, $^{123}$Cd, $^{125}$Cd and $^{133}$Te, for $1/2^-$ isomers in $^{129}$In and $^{131}$In, and for $7^-$ isomers in $^{130}$Sn and $^{134}$Sb. These first direct mass measurements of the Cd and In isomers reveal deviations to the excitation energies based on results from beta-decay experiments and yield new information on neutron- and proton-hole states close to $^{132}$Sn. A new excitation energy of 144(4) keV has been determined for $^{123}$Cd$^m$. A good agreement with the precisely known excitation energies of $^{121}$Cd$^m$, $^{130}$Sn$^m$, and $^{134}$Sb$^m$ has been found.