Recently, angle-resolved photoemission spectroscopy (ARPES) has been used to highlight an anomalously large band renormalization at high binding energies in cuprate superconductors: the high energy waterfall or high energy anomaly (HEA). This paper demonstrates, using a combination of new ARPES measurements and quantum Monte Carlo simulations, that the HEA is not simply the by-product of matrix element effects, but rather represents a cross-over from a quasiparticle band at low binding energies near the Fermi level to valence bands at higher binding energy, assumed to be of strong oxygen character, in both hole- and electron-doped cuprates. While photoemission matrix elements clearly play a role in changing the aesthetic appearance of the band dispersion, i.e. the waterfall-like behavior, they provide an inadequate description for the physics that underlies the strong band renormalization giving rise to the HEA. Model calculations of the single-band Hubbard Hamiltonian showcase the role played by correlations in the formation of the HEA and uncover significant differences in the HEA energy scale for hole- and electron-doped cuprates. In addition, this approach properly captures the transfer of spectral weight accompanying both hole and electron doping in a correlated material and provides a unifying description of the HEA across both sides of the cuprate phase diagram.