If neutrinos get mass via the seesaw mechanism the mixing matrix describing neutrino oscillations can be effectively non-unitary. We show that in this case the neutrino appearance probabilities involve a new CP phase, phi, associated to non-unitarity. This leads to an ambiguity in extracting the standard three--neutrino phase delta_CP, which can survive even after neutrino and antineutrino channels are combined. Its existence should be taken into account in the planning of any oscillation experiment aiming at a robust measurement of delta_CP.