We demonstrate fast initialization of a single hole spin captured in an InGaAs quantum dot with a fidelity F>99% by applying a magnetic field parallel to the growth direction. We show that the fidelity of the hole spin, prepared by ionization of a photo-generated electron-hole pair, is limited by the precession of the exciton spin due to the anisotropic exchange interaction.