Three classes of harmonic disorder systems (Lennard-Jones like glasses, percolators above threshold, and spring disordered lattices) have been numerically investigated in order to clarify the effect of different types of disorder on the mechanism of high frequency sound attenuation. We introduce the concept of frustration in structural glasses as a measure of the internal stress, and find a strong correlation between the degree of frustration and the exponent alpha that characterizes the momentum dependence of the sound attenuation $Gamma(Q)$$simeq$$Q^alpha$. In particular, alpha decreases from about d+1 in low-frustration systems (where d is the spectral dimension), to about 2 for high frustration systems like the realistic glasses examined.