We present broad-band, multi-epoch X-ray spectroscopy of the pulsating ultra-luminous X-ray source (ULX) in NGC 5907. Simultaneous XMM-Newton and NuSTAR data from 2014 are best described by a multi-color black-body model with a temperature gradient as a function of accretion disk radius significantly flatter than expected for a standard thin accretion disk (T(r) ~ r^{-p}, with p=0.608^{+0.014}_{-0.012}). Additionally, we detect a hard power-law tail at energies above 10 keV, which we interpret as being due to Comptonization. We compare this observation to archival XMM-Newton, Chandra, and NuSTAR data from 2003, 2012, and 2013, and investigate possible spectral changes as a function of phase over the 78d super-orbital period of this source. We find that observations taken around phases 0.3-0.4 show very similar temperature profiles, even though the observed flux varies significantly, while one observation taken around phase 0 has a significantly steeper profile. We discuss these findings in light of the recent discovery that the compact object is a neutron star and show that precession of the accretion disk or the neutron star can self-consistently explain most observed phenomena.