A comparative study of the neutron-$gamma$ Pulse Shape Discrimination (PSD) with seven organic scintillators is performed using an identical setup and digital electronics. The scintillators include plastics (EJ-299-33 and a plastic prototype), single crystals (stilbene and the recent doped $p$-terphenyl) and liquids (BC501A, NE213 and the deuterated liquid BC537). First, the overall PSD performance of the different scintillators is compared and threshold neutron energies for a given discrimination quality are determined. Then, using statistical arguments, two intrinsic contributions to the PSD capability of the scintillating materials are disentangled: the light yield and the specific pulse shapes induced by neutrons and $gamma$-rays. This separation provides additional insight into the behaviour of organic scintillators and allows a detailed comparison of the discrimination performance of the various materials. On the basis of this analysis, limitations of current organic scintillators and of recently proposed alternative scintillators are discussed.