Ambient neutrons are one of the most serious backgrounds for underground experiments in search of rare events. The ambient neutron flux in an underground laboratory of Kamioka Observatory was measured using a $mathrm{^3He}$ proportional counter with various moderator setups. Since the detector response largely depends on the spectral shape, the energy spectra of the neutrons transported from the rock to the laboratory were estimated by Monte-Carlo simulations. The ratio of the thermal neutron flux to the total neutron flux was found to depend on the thermalizing efficiency of the rock. Thus, the ratio of the count rate without a moderator to that with a moderator was used to determine this parameter. Consequently, the most-likely neutron spectrum predicted by the simulations for the parameters determined by the experimental results was obtained. The result suggests an interesting spectral shape, which has not been indicated in previous studies. The total ambient neutron flux is $(23.5 pm 0.7 mathrm{_{stat.}} ^{+1.9}_{-2.1} mathrm{_{sys.}}) times 10^{-6}$ cm$^{-2}$ s$^{-1}$. In this paper, we explain our method of the result and discuss our future plan.