Exciton spin relaxation is investigated in single epitaxially grown semiconductor quantum dots in order to test the expected spin relaxation quenching in this system. We study the polarization anisotropy of the photoluminescence signal emitted by isolated quantum dots under steady-state or pulsed non-resonant excitation. We find that the longitudinal exciton spin relaxation time is strikingly short ($leq$100 ps) even at low temperature. This result breaks down the picture of a frozen exciton spin in quantum dots.