Black holes (BHs) play a central role in physics. However, gathering observational evidence for their existence is a notoriously difficult task. Current strategies to quantify the evidence for BHs all boil down to looking for signs of highly compact, horizonless bodies. Here, we study particle creation by objects which collapse to form ultra-compact configurations, with surface at an areal radius $R=R_{f}$ satisfying $1-(2M/R_{f})= epsilon^{2}ll 1$ with $M$ the object mass. We assume that gravitational collapse proceeds in a `standard manner until $R=R_{f}+2M epsilon^{2beta}$, where $beta>0$, and then slows down to form a static object of radius $R_{f}$. In the standard collapsing phase, Hawking-like thermal radiation is emitted, which is as strong as the Hawking radiation of a BH with the same mass but lasts only for $sim 40~(M/M_{odot})[44+ln (10^{-19}/epsilon)]~mu mbox{s}$. Thereafter, in a very large class of models, there exist two bursts of radiation separated by a very long dormant stage. The first burst occurs at the end of the transient Hawking radiation, and is followed by a quiescent stage which lasts for $sim 6times 10^{6}~(epsilon/10^{-19})^{-1}(M/M_{odot})~mbox{yr}$. Afterwards, the second burst is triggered, after which there is no more particle production and the star is forever dark. In a model with $beta=1$, both the first and second bursts outpower the transient Hawking radiation by a factor $sim 10^{38}(epsilon/10^{-19})^{-2}$.