Connecting the observed rest-ultraviolet (UV) luminosities of high-$z$ galaxies to their intrinsic luminosities (and thus star formation rates) requires correcting for the presence of dust. We bypass a common dust-correction approach that uses empirical relationships between infrared (IR) emission and UV colours, and instead augment a semi-empirical model for galaxy formation with a simple -- but self-consistent -- dust model and use it to jointly fit high-$z$ rest-UV luminosity functions (LFs) and colour-magnitude relations ($M_{mathrm{UV}}$-$beta$). In doing so, we find that UV colours evolve with redshift (at fixed UV magnitude), as suggested by observations, even in cases without underlying evolution in dust production, destruction, absorption, or geometry. The observed evolution in our model arises due to the reduction in the mean stellar age and rise in specific star formation rates with increasing $z$. The UV extinction, $A_{mathrm{UV}}$, evolves similarly with redshift, though we find a systematically shallower relation between $A_{mathrm{UV}}$ and $M_{mathrm{UV}}$ than that predicted by IRX-$beta$ relationships derived from $z sim 3$ galaxy samples. Finally, assuming that high $1600 r{A}$ transmission ($gtrsim 0.6$) is a reliable LAE indicator, modest scatter in the effective dust surface density of galaxies can explain the evolution both in $M_{mathrm{UV}}$-$beta$ and LAE fractions. These predictions are readily testable by deep surveys with the James Webb Space Telescope.