In the heavily hole-doped iron-based superconductors $A$Fe$_2$As$_2$ ($A=$ K, Rb, and Cs), the electron effective mass increases rapidly with alkali-ion radius. To study how the mass enhancement affects the superconducting state, we measure the London penetration depth $lambda(T)$ in clean crystals of $A$Fe$_2$As$_2$ down to low temperature $Tsim0.1$ K. In all systems, the superfluid stiffness $rho_s(T)=lambda^2(0)/lambda^2(T)$ can be approximated by a power-law $T$ dependence at low temperatures, indicating the robustness of strong momentum anisotropy in the superconducting gap $Delta(k)$. The power $alpha$ increases from $sim1$ with mass enhancement and approaches an unconventional exponent $alphasim 1.5$ in the heaviest CsFe$_2$As$_2$. This appears to be a hallmark of superconductors near antiferromagnetic quantum critical points, where the quasiparticles excited across the anisotropic $Delta(k)$ are significantly influenced by the momentum dependence of quantum critical fluctuations.