Superconducting MgB$_2$ shows an E$_{2g}$ zone center phonon, as measured by Raman spectroscopy, that is very broad in energy and temperature dependent. The Raman shift and lifetime show large differences with the values elsewhere in the Brillouin Zone measured by Inelastic X-ray Scattering (IXS), where its dispersion can be accounted for by standard harmonic phonon theory, adding only a moderate electron-phonon coupling. Here we show that the effects rapidly disappear when electron-phonon coupling is switched off by Al substitution on the Mg sites. Moreover, using IXS with very high wave-vector resolution in MgB$_2$, we can follow the dispersion connecting the Raman and the IXS signal, in agreement with a theory using only electron-phonon coupling but without strong anharmonic terms. The observation is important in order to understand the effects of electron-phonon coupling on zone center phonons modes in MgB$_2$, but also in all metals characterized by a small Fermi velocity in a particular direction, typical for layered compounds.