We report an investigation of the lattice dynamical properties in a range of Fe$_{1+y}$Te$_{1-x}$Se$_{x}$ compounds, with special emphasis on the c-axis polarized vibration of Fe with B$_{1g}$ symmetry, a Raman active mode common to all families of Fe-based superconductors. We have carried out a systematic study of the temperature dependence of this phonon mode as a function of Se $x$ and excess Fe $y$ concentrations. In parent compound Fe$_{1+y}$Te, we observe an unconventional broadening of the phonon between room temperature and magnetic ordering temperature $T_N$. The situation smoothly evolves towards a regular anharmonic behavior as Te is substituted for Se and long range magnetic order is replaced by superconductivity. Irrespective to Se contents, excess Fe is shown to provide an additional damping channel for the B$_{1g}$ phonon at low temperatures. We performed Density Functional Theory (DFT) ab-initio calculations within the local density approximation (LDA) to calcuate the phonon frequencies including magnetic polarization and Fe non-stoichiometry in the Virtual Crystal Approximation (VCA). We obtained a good agreement with the measured phonon frequencies in the Fe-deficient samples, while the effects of Fe excess are poorly reproduced. This may be due to excess Fe-induced local magnetism and low energy magnetic fluctuations that can not be treated accurately within these approaches. As recently revealed by neutron scattering and $mu$-SR studies, these phenomena occur in the temperature range where anomalous decay of the B$_{1g}$ phonon is observed, and suggests a peculiar coupling of this mode with local moments and spin fluctuations in Fe$_{1+y}$Te$_{1-x}$Se$_{x}$.