We report in situ neutron background measurements at the Kuo-Sheng Reactor Neutrino Laboratory (KSNL) by a hybrid neutron detector (HND) with a data size of 33.8 days under identical shielding configurations as during the neutrino physics data taking. The HND consists of BC-501A liquid and BC-702 phosphor powder scintillation neutron detectors, which is sensitive to both fast and thermal neutrons, respectively. Neutron-induced events for the two channels are identified and differentiated by pulse shape analysis, such that background of both are simultaneously measured. The fast neutron fluxes are derived by an iterative unfolding algorithm. Neutron induced background in the germanium detector under the same fluxes, both due to cosmic-rays and ambient radioactivity, are derived and compared with the measurements. The results are valuable to background understanding of the neutrino data at the KSNL. In particular, neutron-induced background events due to ambient radioactivity as well as from reactor operation are negligible compared to intrinsic cosmogenic activity and ambient $gamma$-activity. The detector concept and analysis procedures are applicable to neutron background characterization in similar rare-event experiments.