It is currently thought that the early dark energy (EDE) resolution of the Hubble tension will inevitably suffer inconsistency with the large scale structure data (quantified as $S_8$). However, if this so-called $S_8$ tension is physical, it might be related only with the clustering property of dark matter at the corresponding scale. We find by performing Monte Carlo Markov Chain analysis that in the AdS-EDE model (with an Anti-de Sitter phase around recombination), if an axion field with mass $m_asimeq1.3times10^{-26}$ eV becomes dynamical at redshift $zsimeq 1.7times10^4$ and constitutes $7%$ of the total dark matter, both $H_0$ and $S_8$ will be consistent with local measurements within $1sigma$, while the model can fit PlanckCMB+SN+BAO+EFT dataset as well as $Lambda$CDM, which will possibly be tested with on-going CMB and galaxy surveys.