Fast hadrons have been observed to cause a cumulative damage in Lead Tungstate and LYSO crystals. The underlying mechanism has been proven to be the creation of fission tracks, which act as scattering centres, thus reducing the light collection efficiency. For calorimetry applications in an environment where large, fast hadron fluences are anticipated, predictions about damage in crystals are of great importance for making an informed choice of technology. In the study presented here, simulations using the FLUKA package have been performed on Lead Tungstate, LYSO and Cerium Fluoride, and their results have been compared with measurements. The agreement that is found between simulation results and experimental measurements allows to conclude that the damage amplitude in a given material can be predicted with a precision that is sufficient to anticipate the damage expected during detector operation.