Charged-particle spectra associated with direct photon ($gamma_{dir} $) and $pi^0$ are measured in $p$+$p$ and Au+Au collisions at center-of-mass energy $sqrt{s_{_{NN}}}=200$ GeV with the STAR detector at RHIC. A hower-shape analysis is used to partially discriminate between $gamma_{dir}$ and $pi^0$. Assuming no associated charged particles in the $gamma_{dir}$ direction (near side) and small contribution from fragmentation photons ($gamma_{frag}$), the associated charged-particle yields opposite to $gamma_{dir}$ (away side) are extracted. At mid-rapidity ($|eta|<0.9$) in central Au+Au collisions, charged-particle yields associated with $gamma_{dir}$ and $pi^0$ at high transverse momentum ($8< p_{T}^{trig}<16$ GeV/$c$) are suppressed by a factor of 3-5 compared with $p$ + $p$ collisions. The observed suppression of the associated charged particles, in the kinematic range $|eta|<1$ and $3< p_{T}^{assoc} < 16$ GeV/$c$, is similar for $gamma_{dir}$ and $pi^0$, and independent of the $gamma_{dir}$ energy within uncertainties. These measurements indicate that the parton energy loss, in the covered kinematic range, is insensitive to the parton path length.