Rare information on photodisintegration reactions of nuclei with mass numbers $A approx 160$ at astrophysical conditions impedes our understanding of the origin of $p$-nuclei. Experimental determination of the key ($p,gamma$) cross sections has been playing an important role to verify nuclear reaction models and to provide rates of relevant ($gamma,p$) reactions in $gamma$-process. In this paper we report the first cross section measurements of $^{160}$Dy($p,gamma$)$^{161}$Ho and $^{161}$Dy($p,n$)$^{161}$Ho in the beam energy range of 3.4 - 7.0 MeV, partially covering the Gamow window. Such determinations are possible by using two targets with various isotopic fractions. The cross section data can put a strong constraint on the nuclear level densities and gamma strength functions for $A approx$ 160 in the Hauser-Feshbach statistical model. Furthermore, we find the best parameters for TALYS that reproduce the A $thicksim$ 160 data available, $^{160}$Dy($p,gamma$)$^{161}$Ho and $^{162}$Er($p,gamma$)$^{163}$Tm, and recommend the constrained $^{161}$Ho($gamma,p$)$^{160}$Dy reaction rates over a wide temperature range for $gamma$-process network calculations. Although the determined $^{161}$Ho($gamma$, p) stellar reaction rates at the temperature of 1 to 2 GK can differ by up to one order of magnitude from the NON-SMOKER predictions, it has a minor effect on the yields of $^{160}$Dy and accordingly the $p$-nuclei, $^{156,158}$Dy. A sensitivity study confirms that the cross section of $^{160}$Dy($p$, $gamma$)$^{161}$Ho is measured precisely enough to predict yields of $p$-nuclei in the $gamma$-process.