We study the dependence of the vacuum Rabi splitting (VRS) on frequency disorder, vibrations, near-field effects and density in molecular polaritonics. In the mesoscopic limit, static frequency disorder alone can already introduce a loss mechanism from polaritonic states into a dark state reservoir, which we quantitatively describe, providing an analytical scaling of the VRS with the level of disorder. Disorder additionally can split a molecular ensemble into donor-type and acceptor-type molecules and the combination of vibronic coupling, dipole-dipole interactions and vibrational relaxation induces an incoherent FRET (F{o}rster resonance energy transfer) migration of excitations within the collective molecular state. This is equivalent to a dissipative disorder and has the effect of saturating and even reducing the VRS in the mesoscopic, high-density limit. Overall, this analysis allows to quantify the crucial role played by dark states in cavity quantum electrodynamics with mesoscopic, disordered ensembles.