Future communication and computation technologies that exploit quantum information require robust and well-isolated qubits. Electron spins in III-V semiconductor quantum dots, while promising candidates, see their dynamics limited by undesirable hysteresis and decohering effects of the nuclear spin bath. Replacing electrons with holes should suppress the hyperfine interaction and consequently eliminate strong nuclear effects. Using picosecond optical pulses, we demonstrate coherent control of a single hole qubit and examine both free-induction and spin-echo decay. In moving from electrons to holes, we observe significantly reduced hyperfine interactions, evidenced by the reemergence of hysteresis-free dynamics, while obtaining similar coherence times, limited by non-nuclear mechanisms. These results demonstrate the potential of optically controlled, quantum dot hole qubits.