We investigate the origin of imperfections in the fidelity of a two-photon controlled-phase gate based on two-level-emitter non-linearities. We focus on a passive system that operates without external modulations to enhance its performance. We demonstrate that the fidelity of the gate is limited by opposing requirements on the input pulse width for one- and two-photon scattering events. For one-photon scattering, the spectral pulse width must be narrow compared to the emitter linewidth, while two-photon scattering processes require the pulse width and emitter linewidth to be comparable. We find that these opposing requirements limit the maximum fidelity of the two-photon controlled-phase gate for Gaussian photon pulses to 84%.