Charge transport layers (CTLs) are key components of diffusion controlled perovskite solar cells, however, they can induce additional non-radiative recombination pathways which limit the open circuit voltage (V_OC) of the cell. In order to realize the full thermodynamic potential of the perovskite absorber, both the electron and hole transport layer (ETL/HTL) need to be as selective as possible. By measuring the quasi-Fermi level splitting (QFLS) of perovskite/CTL heterojunctions, we quantify the non-radiative interfacial recombination current for a wide range of commonly used CTLs, including various hole-transporting polymers, spiro-OMeTAD, metal oxides and fullerenes. We find that all studied CTLs limit the V_OC by inducing an additional non-radiative recombination current that is significantly larger than the loss in the neat perovskite and that the least-selective interface sets the upper limit for the V_OC of the device. The results also show that the V_OC equals the internal QFLS in the absorber layer of (pin, nip) cells with selective CTLs and power conversion efficiencies of up to 21.4%. However, in case of less selective CTLs, the V_OC is substantially lower than the QFLS which indicates additional losses at the contacts and/or interfaces. The findings are corroborated by rigorous device simulations which outline several important considerations to maximize the V_OC. This work shows that the real challenge to supress non-radiative recombination losses in perovskite cells on their way to the radiative limit lies in the suppression of carrier recombination at the perovskite/CTL interfaces.