The large radii of many hot Jupiters can only be matched by models that have hot interior adiabats, and recent theoretical work has shown that the interior evolution of hot Jupiters has a significant impact on their atmospheric structure. Due to its inflated radius, low gravity, and ultra-hot equilibrium temperature, WASP-76b is an ideal case study for the impact of internal evolution on observable properties. Hot interiors should most strongly affect the non-irradiated side of the planet, and thus full phase curve observations are critical to ascertain the effect of the interior on the atmospheres of hot Jupiters. In this work, we present the first Spitzer phase curve observations of WASP-76b. We find that WASP-76b has an ultra-hot day side and relatively cold nightside with brightness temperatures of $2471 pm 27~mathrm{K}$/$1518 pm 61~mathrm{K}$ at $3.6~micron$ and $2699 pm 32~mathrm{K}$/$1259 pm 44~mathrm{K}$ at $4.5~micron$, respectively. These results provide evidence for a dayside thermal inversion. Both channels exhibit small phase offsets of $0.68 pm 0.48^{circ}$ at $3.6~micron$ and $0.67 pm 0.2^{circ}$ at $4.5~mumathrm{m}$. We compare our observations to a suite of general circulation models that consider two end-members of interior temperature along with a broad range of frictional drag strengths. Strong frictional drag is necessary to match the small phase offsets and cold nightside temperatures observed. From our suite of cloud-free GCMs, we find that only cases with a cold interior can reproduce the cold nightsides and large phase curve amplitude at $4.5~micron$, hinting that the hot interior adiabat of WASP-76b does not significantly impact its atmospheric dynamics or that clouds blanket its nightside.