Energy Recovery Linacs provide high-energy beams, but decelerate those beams before dumping them, so that their energy is available for the acceleration of new particles. During this deceleration, any relative energy spread that is created at high energy is amplified by the ratio between high energy and dump energy. Therefore, Energy Recovery Linacs are sensitive to energy spread acquired at high energy, e.g. from wake fields. One can compensate the time-correlated energy spread due to wakes via energy-dependent time-of-flight terms in appropriate sections of an Energy Recovery Linac, and via high-frequency cavities. We show that nonlinear time-of-flight terms can only eliminate odd orders in the correlation between time and energy, if these terms are created by a beam transport within the linac that is common for accelerating and decelerating beams. If these two beams are separated, so that different beam transport sections can be used to produce time-of-flight terms suitable for each, also even-order terms in the energy spread can be eliminated. As an example, we investigate the potential of using this method for the Cornell x-ray Energy Recovery Linac. Via quadratic time-of-flight terms, the energy spread can be reduced by 66%. Alternatively, since the energy spread from the dominantly resistive wake fields of the analysed accelerator is approximately harmonic in time, a high-frequency cavity could diminish the energy spread by 81%. This approach would require bunch-lengthening and recompression in separate sections for accelerating and decelerating beams. Such sections have therefore been included in Cornells x-ray Energy Recovery Linac design.