We analyze the detection of itinerant photons using a quantum non-demolition (QND) measurement. We show that the backaction due to the continuous measurement imposes a limit on the detector efficiency in such a scheme. We illustrate this using a setup where signal photons have to enter a cavity in order to be detected dispersively. In this approach, the measurement signal is the phase shift imparted to an intense beam passing through a second cavity mode. The restrictions on the fidelity are a consequence of the Quantum Zeno effect, and we discuss both analytical results and quantum trajectory simulations of the measurement process.