Optically induced ultrafast switching of single photons is demonstrated by rotating the photon polarization via the Kerr effect in a commercially available single mode fiber. A switching efficiency of 97% is achieved with a $sim1.7$,ps switching time, and signal-to-noise ratio of $sim800$. Preservation of the quantum state is confirmed by measuring no significant increase in the second-order autocorrelation function $g^{(2)}(0)$. These values are attained with only nanojoule level pump energies that are produced by a laser oscillator with 80,MHz repetition rate. The results highlight a simple switching device capable of both high-bandwidth operations and preservation of single-photon properties for applications in photonic quantum processing and ultrafast time-gating or switching.