We consider a hybrid single-electron transistor (SET) constituted by a gate-controlled normal-metal island (N) connected to two voltage-biased superconducting leads (S) by means of two tunnel junctions (S-I-N-I-S), operated as a turnstile. We show that the exchange of photons between this system and the high-temperature electromagnetic environment where it is embedded enhances Andreev reflection, thereby limiting the single-electron tunneling accuracy.