We have measured the carrier spin dynamics in p-doped InAs/GaAs quantum dots by pump-probe photo-induced circular dichroism and time-resolved photoluminescence experiments. We show that the hole spin dephasing is controlled by the hyperfine interaction between hole and nuclear spins. In the absence of external magnetic field, we find a characteristic hole spin dephasing time of 15 ns, in close agreement with our calculations based on dipole-dipole coupling between the hole and the quantum dot nuclei. Finally we demonstrate that a small external magnetic field, typically 10 mT instead of 200 mT for the case of electrons, quenches the hyperfine hole spin dephasing.