We have studied neutron response of PARIS phoswich [LaBr$_3$(Ce)-NaI(Tl)] detector which is being developed for measuring the high energy (E$_{gamma}$ = 5 - 30 MeV) $gamma$ rays emitted from the decay of highly collective states in atomic nuclei. The relative neutron detection efficiency of LaBr$_3$(Ce) and NaI(Tl) crystal of the phoswich detector has been measured using the time-of-flight (TOF) and pulse shape discrimination (PSD) technique in the energy range of E$_n$ = 1 - 9 MeV and compared with the GEANT4 based simulations. It has been found that for E$_n$ $>$ 3 MeV, $sim$ 95 % of neutrons have the primary interaction in the LaBr$_3$(Ce) crystal, indicating that a clear n-$gamma$ separation can be achieved even at $sim$15 cm flight path.