We discuss the energy loss due to gravitational radiation of binaries composed of exotic objects whose horizon boundary conditions are replaced with reflective ones. Our focus is on the extreme mass-ratio inspirals, in which the central heavier black hole is replaced with an exotic compact object. We show, in this case, a modulation of the energy loss rate depending on the evolving orbital frequency occurs and leads to two different types of modifications to the gravitational wave phase evolution; the oscillating part directly corresponding to the modulation in the energy flux, and the non-oscillating part coming from the quadratic order in the modulation. This modification can be sufficiently large to detect with future space-borne detectors.