The $^{22}$Ne($p,gamma$)$^{23}$Na reaction, part of the neon-sodium cycle of hydrogen burning, may explain the observed anticorrelation between sodium and oxygen abundances in globular cluster stars. Its rate is controlled by a number of low-energy resonances and a slowly varying non-resonant component. Three new resonances at $E_p$ = 156.2, 189.5, and 259.7 keV have recently been observed and confirmed. However, significant uncertainty on the reaction rate remains due to the non-resonant process and to two suggested resonances at $E_p$ = 71 and 105 keV. Here, new $^{22}$Ne($p,gamma$)$^{23}$Na data with high statistics and low background are reported. Stringent upper limits of 6$times$10$^{-11}$ and 7$times$10$^{-11}$,eV (90% confidence level), respectively, are placed on the two suggested resonances. In addition, the off-resonant S-factor has been measured at unprecedented low energy, constraining the contributions from a subthreshold resonance and the direct capture process. As a result, at a temperature of 0.1 GK the error bar of the $^{22}$Ne($p,gamma$)$^{23}$Na rate is now reduced by three orders of magnitude.