We determined the lattice dynamics of metastable, surface-stabilized $alpha$-phase FeSi$_2$ nanoislands epitaxially grown on the Si(111) surface with average heights and widths ranging from 1.5 to 20 nm and 18 to 72 nm, respectively. The crystallographic orientation, surface morphology and local crystal structure of the nanoislands were investigated by reflection high-energy electron diffraction, atomic force microscopy and X-ray absorption spectroscopy. The Fe-partial phonon density of states (PDOS), obtained by nuclear inelastic scattering, exhibits a pronounced damping and broadening of the spectral features with decreasing average island height. First-principles calculations of the polarization-projected Si- and Fe-partial phonon dispersions and PDOS enable the disentanglement of the contribution of the $xy$- and $z$-polarized phonons to the experimental PDOS. Modeling of the experimental data with the theoretical results unveils an enhanced damping of the $z$-polarized phonons for islands with average sizes below 10 nm. This phenomenon is attributed to the fact that the low-energy $z$-polarized phonons couple to the low-energy surface/interface vibrational modes. The thermodynamic and elastic properties obtained from the experimental data show a pronounced size-dependent behavior.