Interatomic Coulombic decay (ICD) is a mechanism which allows microscopic objects to rapidly exchange energy. When the two objects are distant, the energy transfer between the donor and acceptor species takes place via the exchange of a virtual photon. On the contrary, recent ab initio calculations have revealed that the presence of a third passive species can significantly enhance the ICD rate at short distances due to the effects of electronic wave function overlap and charge transfer states [Phys. Rev. Lett. 119, 083403 (2017)]. Here, we develop a virtual photon description of three-body ICD, showing that a mediator atom can have a significant influence at much larger distances. In this regime, this impact is due to the scattering of virtual photons off the mediator, allowing for simple analytical results and being manifest in a distinct geometry-dependence which includes interference effects. As a striking example, we show that in the retarded regime ICD can be substantially enhanced or suppressed depending on the position of the ICD-inactive object, even if the latter is far from both donor and acceptor species.