We report on the measurements of the fluxes and spectra of the environmental fast neutron background at the China Jinping Underground Laboratory (CJPL) with a rock overburden of about 6700 meters water equivalent, using a liquid scintillator detector doped with 0.5% gadolinium. The signature of a prompt nuclear recoil followed by a delayed high energy $gamma$-ray cascade is used to identify neutron events. The large energy deposition of the delayed $gamma$-rays from the $(n, gamma)$ reaction on gadolinium, together with the excellent n-$gamma$ discrimination capability provides a powerful background suppression which allows the measurement of a low intensity neutron flux. The neutron flux of $(1.51pm0.03(stat.)pm0.10(syst.))times10^{-7}$ cm$^{-2}$s$^{-1}$ in the energy range of 1 -- 10 MeV in the Hall A of CJPL was measured based on 356 days of data. In the same energy region, measurement with the same detector placed in a one meter thick polyethylene room gives a significantly lower flux of $(4.9pm0.9(stat.)pm0.5(syst.))times10^{-9}$ cm$^{-2}$s$^{-1}$ with 174 days of data. This represents a measurement of the lowest environmental fast neutron background among the underground laboratories in the world, prior to additional experiment-specific attenuation. Additionally, the fast neutron spectra both in the Hall A and the polyethylene room were reconstructed with the help of GEANT4 simulation.