The mismatch between the locally measured expansion rate of the universe and the one inferred from the cosmic microwave background measurements by Planck in the context of the standard $Lambda$CDM, known as the Hubble tension, has become one of the most pressing problems in cosmology. A large number of amendments to the $Lambda$CDM model have been proposed in order to solve this tension. Many of them introduce new physics, such as early dark energy, modifications of the standard model neutrino sector, extra radiation, primordial magnetic fields or varying fundamental constants, with the aim of reducing the sound horizon at recombination $r_{star}$. We demonstrate here that any model which only reduces $r_{star}$ can never fully resolve the Hubble tension while remaining consistent with other cosmological datasets. We show explicitly that models which achieve a higher Hubble constant with lower values of matter density $Omega_m h^2$ run into tension with the observations of baryon acoustic oscillations, while models with larger $Omega_mh^2$ develop tension with galaxy weak lensing data.