Current interferometric gravitational wave detectors (IGWDs) are operated at room temperature with test masses made from fused silica. Fused silica shows very low absorption at the laser wavelength of 1064 nm. It is also well suited to realize low thermal noise floors in the detector signal band since it offers low mechanical loss, i. e. high quality factors (Q factors) at room temperature. However, for a further reduction of thermal noise, cooling the test masses to cryogenic temperatures may prove an interesting technique. Here we compare the results of Q factor measurements at cryogenic temperatures of acoustic eigenmodes of test masses from fused silica and its crystalline counterpart. Our results show that the mechanical loss of fused silica increases with lower temperature and reaches a maximum at 30 K for frequencies of slightly above 10 kHz. The losses of crystalline quartz generally show lower values and even fall below the room temperature values of fused silica below 10 K. Our results show that in comparison to fused silica, crystalline quartz has a considerably narrower and lower dissipation peak on cooling and thus has more promise as a test mass material for IGDWs operated at cryogenic temperatures. The origin of the different Q factor versus temperature behavior of the two materials is discussed.