In this paper we describe the development and first tests of a neutron spectrometer designed for high flux environments, such as the ones found in fast nuclear reactors. The spectrometer is based on the conversion of neutrons impinging on $^6$Li into $alpha$ and $t$ whose total energy comprises the initial neutron energy and the reaction $Q$-value. The $^6$LiF layer is sandwiched between two CVD diamond detectors, which measure the two reaction products in coincidence. The spectrometer was calibrated at two neutron energies in well known thermal and 3 MeV neutron fluxes. The measured neutron detection efficiency varies from 4.2$times 10^{-4}$ to 3.5$times 10^{-8}$ for thermal and 3 MeV neutrons, respectively. These values are in agreement with Geant4 simulations and close to simple estimates based on the knowledge of the $^6$Li(n,$alpha$)$t$ cross section. The energy resolution of the spectrometer was found to be better than 100 keV when using 5 m cables between the detector and the preamplifiers.