Deterministic quantum interactions between single photons and single quantum emitters are a vital building block towards the distribution of quantum information between remote systems. Deterministic photon-atom state transfer has been demonstrated by using protocols that include active feedback or synchronized control pulses. Here we demonstrate a completely passive swap gate between the states of a single photon and a single atom. The underlying mechanism is single-photon Raman interaction (SPRINT) - an interference-based effect in which a photonic qubit deterministically controls the state of a material qubit encoded in the two ground states of a {Lambda} system, and vice versa. Using a nanofiber-coupled microsphere resonator coupled to single Rb atoms we swap a photonic qubit into the atom and back, demonstrating nonclassical fidelities in both directions. Requiring no control fields or feedback protocol, the gate takes place automatically at the timescale of the atoms cavity- enhanced spontaneous emission time. Applicable to any waveguide-coupled {Lambda} system, this scheme provides a versatile building block for the modular scaling up of quantum information processing systems.