We present measurements of the size of the quasar proximity zone ($R_p$) for eleven low-luminosity ($-26.16leq M_{1450}leq-22.83$) quasars at $zsim6$, discovered by the Subaru High-$z$ Exploration of Low-Luminosity Quasars project (SHELLQs). Our faint quasar sample expands the $R_p$ measurement down to $M_{1450}=-22.83$ mag, where more common quasar populations dominate at the epoch. We restrict the sample to quasars whose systemic redshifts have been precisely measured by [CII] 158 $mu$m or MgII $lambda$2798 emission lines. We also update the $R_p$ measurements for 26 luminous quasars presented in Eilers et al. (2017)(arXiv:1703.02539) by using the latest systemic redshift results. The luminosity dependence on $R_p$ is found to be consistent with the theoretical prediction assuming highly ionized intergalactic medium. We find a shallow redshift evolution of the luminosity-corrected $R_p$, $R_{p,{rm corr}}^{-25}$ ($R_{p, rm corr}^{-25}propto(1+z)^{-3.79pm1.72}$) over $5.8lesssim z lesssim6.6$. This trend is steeper than that of Eilers et al. (2017) but significantly shallower than those of the earlier studies. Our results suggest that $R_{p,rm corr}$ is insensitive to the neutral fraction of the universe at $zsim6$. Four quasars show exceptionally small $R_{p,rm corr}^{-25}$ ($ lesssim0.90$ proper Mpc), which could be the result of their young age ($<10^4$ yr) in the reionization epoch, though statistics is still small.