The spin interaction of a hole confined in a quantum dot with the surrounding nuclei is described in terms of an effective magnetic field. We show that, in contrast to the Fermi contact hyperfine interaction for conduction electrons, the dipole-dipole hyperfine interaction is anisotropic for a hole, for both pure or mixed hole states. We evaluate the coupling constants of the hole-nuclear interaction and demonstrate that they are only one order of magnitude smaller than the coupling constants of the electron-nuclear interaction. We also study, theoretically, the hole spin dephasing of an ensemble of quantum dots via the hyperfine interaction in the framework of frozen fluctuations of the nuclear field, in absence or in presence of an applied magnetic field. We also discuss experiments which could evidence the dipole-dipole hyperfine interaction and give information on hole mixing.