Gamma-ray bursts (GRBs) offer a route to characterizing star-forming galaxies and quantifying high-$z$ star formation that is distinct from the approach of traditional galaxy surveys: GRB selection is independent of dust and probes even the faintest galaxies that can evade detection in flux-limited surveys. However, the exact relation between the GRB rate and the star formation rate (SFR) throughout all redshifts is controversial. The Optically Unbiased GRB Host (TOUGH) survey includes observations of all GRB hosts (69) in an optically unbiased sample of Swift GRBs and we utilize these to constrain the evolution of the UV GRB-host-galaxy luminosity function (LF) between $z=0$ and $z=4.5$, and compare this with LFs derived from both Lyman-break galaxy (LBG) surveys and simulation modeling. At all redshifts we find the GRB hosts to be most consistent with a luminosity function derived from SFR weighted models incorporating GRB production via both metallicity-dependent and independent channels with a relatively high level of bias toward low metallicity hosts. In the range $1<z<3$ an SFR weighted LBG derived (i.e., non-metallicity biased) LF is also a reasonable fit to the data. Between $zsim3$ and $zsim6$, we observe an apparent lack of UV bright hosts in comparison with LBGs, though the significance of this shortfall is limited by nine hosts of unknown redshift.