We utilise a series of high-resolution cosmological zoom simulations of galaxy formation to investigate the relationship between the ultraviolet (UV) slope, beta, and the ratio of the infrared luminosity to UV luminosity (IRX) in the spectral energy distributions (SEDs) of galaxies. We employ dust radiative transfer calculations in which the SEDs of the stars in galaxies propagate through the dusty interstellar medium. Our main goals are to understand the origin of, and scatter in the IRX-beta relation; to assess the efficacy of simplified stellar population synthesis screen models in capturing the essential physics in the IRX-beta relation; and to understand systematic deviations from the canonical local IRX-beta relations in particular populations of high-redshift galaxies. Our main results follow. Galaxies that have young stellar populations with relatively cospatial UV and IR emitting regions and a Milky Way-like extinction curve fall on or near the standard Meurer relation. This behaviour is well captured by simplified screen models. Scatter in the IRX-beta relation is dominated by three major effects: (i) older stellar populations drive galaxies below the relations defined for local starbursts due to a reddening of their intrinsic UV SEDs; (ii) complex geometries in high-z heavily star forming galaxies drive galaxies toward blue UV slopes owing to optically thin UV sightlines; (iii) shallow extinction curves drive galaxies downward in the IRX-beta plane due to lowered NUV/FUV extinction ratios. We use these features of the UV slopes of galaxies to derive a fitting relation that reasonably collapses the scatter back toward the canonical local relation. Finally, we use these results to develop an understanding for the location of two particularly enigmatic populations of galaxies in the IRX-beta plane: z~2-4 dusty star forming galaxies, and z>5 star forming galaxies.