Silicates are an important component of interstellar dust and the structure of these grains -- amorphous versus crystalline -- is sensitive to the local physical conditions. We have studied the infrared spectra of a sample of ultra-luminous infrared galaxies. Here, we report the discovery of weak, narrow absorption features at 11, 16, 19, 23, and 28 microns, characteristic of crystalline silicates, superimposed on the broad absorption bands at 10 and 18 microns due to amorphous silicates in a subset of this sample. These features betray the presence of forsterite (Mg_2SiO_4), the magnesium-rich end member of the olivines. Previously, crystalline silicates have only been observed in circumstellar environments. The derived fraction of forsterite to amorphous silicates is typically 0.1 in these ULIRGs. This is much larger than the upper limit for this ratio in the interstellar medium of the Milky Way, 0.01. These results suggest that the timescale for injection of crystalline silicates into the ISM is short in a merger-driven starburst environment (e.g., as compared to the total time to dissipate the gas), pointing towards massive stars as a prominent source of crystalline silicates. Furthermore, amorphization due to cosmic rays, which is thought to be of prime importance for the local ISM, lags in vigorous starburst environments.