We investigate the behavior of the return amplitude ${cal F}(t)= |langlePsi(0)|Psi(t)rangle|$ following a quantum quench in a conformal field theory (CFT) on a compact spatial manifold of dimension $d-1$ and linear size $O(L)$, from a state $|Psi(0)rangle$ of extensive energy with short-range correlations. After an initial gaussian decay ${cal F}(t)$ reaches a plateau value related to the density of available states at the initial energy. However for $d=3,4$ this value is attained from below after a single oscillation. For a holographic CFT the plateau persists up to times at least $O(sigma^{1/(d-1)} L)$, where $sigmagg1$ is the dimensionless Stefan-Boltzmann constant. On the other hand for a free field theory on manifolds with high symmetry there are typically revivals at times $tsimmbox{integer}times L$. In particular, on a sphere $S_{d-1}$ of circumference $2pi L$, there is an action of the modular group on ${cal F}(t)$ implying structure near all rational values of $t/L$, similarly to what happens for rational CFTs in $d=2$.