We present direct photon-hadron correlations in 200 GeV/A Au$+$Au, $d$$+$Au and $p$$+$$p$ collisions, for direct photon $p_T$ from 5--12 GeV/$c$, collected by the PHENIX Collaboration in the years from 2006 to 2011. We observe no significant modification of jet fragmentation in $d$$+$Au collisions, indicating that cold nuclear matter effects are small or absent. Hadrons carrying a large fraction of the quarks momentum are suppressed in Au$+$Au compared to $p$$+$$p$ and $d$$+$Au. As the momentum fraction decreases, the yield of hadrons in Au$+$Au increases to an excess over the yield in $p$$+$$p$ collisions. The excess is at large angles and at low hadron $p_T$ and is most pronounced for hadrons associated with lower momentum direct photons. Comparison to theoretical calculations suggests that the hadron excess arises from medium response to energy deposited by jets.