We detect 20 $z=7.0$ Ly$alpha$ emitter (LAE) candidates to $L({rm Ly}alpha) geq 2 times 10^{42}$ erg s$^{-1}$ or $0.3$ $L^*_{z=7}$ and in $6.1times 10^5$ Mpc$^3$ volume in the Subaru Deep Field and the Subaru/XMM-Newton Deep Survey field by 82 and 37 hours of Subaru Suprime-Cam narrowband NB973 and reddest optical $y$-band imaging. We compare their Ly$alpha$ and UV luminosity functions (LFs) and densities and Ly$alpha$ equivalent widths (EWs) to those of $z=5.7$, 6.6 and 7.3 LAEs from previous Suprime-Cam surveys. The Ly$alpha$ LF (density) rapidly declines by a factor of $times$1.5 (1.9) in $L({rm Ly}alpha)$ at $z=5.7-6.6$ (160 Myr), $times$1.5 (1.6) at $z=6.6-7.0$ (60 Myr) at the faint end and $times$2.0 (3.8) at $z=7.0-7.3$ (40 Myr). Also, in addition to the systematic decrease in EW at $z=5.7-6.6$ previously found, 2/3 of the $z=7.0$ LAEs detected in the UV continuum exhibit lower EWs than the $z=6.6$ ones. Moreover, while the UV LF and density do not evolve at $z=5.7-6.6$, they modestly decline at $z=6.6-7.0$, implying galaxy evolution contributing to the decline of the Ly$alpha$ LF. Comparison of the $z=7.0$ Ly$alpha$ LF to the one predicted by an LAE evolution model further reveals that galaxy evolution alone cannot explain all the decline of Ly$alpha$ LF. If we attribute the discrepancy to Ly$alpha$ attenuation by neutral hydrogen, the intergalactic medium transmission of Ly$alpha$ photons at $z=7.0$ would be $T_{{rm Ly}alpha}^{rm IGM} leq 0.6-0.7$. It is lower (higher) than $T_{{rm Ly}alpha}^{rm IGM}$ at $z=6.6$ (7.3) derived by previous studies, suggesting rapid increase in neutral fraction at $z > 6$.