Tidal Disruption Events (TDEs) are transient events observed when a star passes close enough to a supermassive black hole to be tidally destroyed. Many TDE candidates have been discovered in host galaxies whose spectra have weak or no line emission yet strong Balmer line absorption, indicating a period of intense star formation that has recently ended. As such, TDE host galaxies fall into the rare class of quiescent Balmer-strong galaxies. Here, we quantify the fraction of galaxies in the Sloan Digital Sky Survey (SDSS) with spectral properties like those of TDE hosts, determining the extent to which TDEs are over-represented in such galaxies. Galaxies whose spectra have Balmer absorption H$delta_{rm A}$ $-$ $sigma$(H$delta_{rm A}$) $>$ 4 AA (where $sigma$(H$delta_{rm A}$) is the error in the Lick H$delta_{rm A}$ index) and H$alpha$ emission EW $<$ $3$ AA have had a strong starburst in the last $sim$Gyr. They represent 0.2% of the local galaxy population, yet host 3 of 8 (37.5%) optical/UV-selected TDE candidates. A broader cut, H$delta_{rm A} >$ 1.31 AA and H$alpha$ EW $<$ $3$ AA, nets only 2.3% of SDSS galaxies, but 6 of 8 (75%) optical/UV TDE hosts. Thus, quiescent Balmer-strong galaxies are over-represented among the TDE hosts by a factor of 33-190. The high-energy-selected TDE Swift J1644 also lies in a galaxy with strong Balmer lines and weak H$alpha$ emission, implying a $>80times$ enhancement in such hosts and providing an observational link between the $gamma$/X-ray-bright and optical/UV-bright TDE classes.