We propose a mechanism of energy relaxation for carriers confined in a non-polar quantum dot surrounded by an amorphous polar environment. The carrier transitions are due to their interaction with the oscillating electric field induced by the local vibrations in the surrounding amorphous medium. We demonstrate that this mechanism controls energy relaxation for electrons in Si nanocrystals embedded in a SiO$_2$ matrix, where conventional mechanisms of electron-phonon interaction are not efficient.