We demonstrate how the least luminous galaxies in the Universe, ultra-faint dwarf galaxies, are sensitive to their dynamical mass at the time of cosmic reionization. We select a low-mass ($sim text{1.5} times 10^{9} , text{M}_{odot}$) dark matter halo from a cosmological volume, and perform zoom hydrodynamical simulations with multiple alternative histories using genetically modified initial conditions. Earlier forming ultra-faints have higher stellar mass today, due to a longer period of star formation before their quenching by reionization. Our histories all converge to the same final dynamical mass, demonstrating the existence of extended scatter ($geq$ 1 dex) in stellar masses at fixed halo mass due to the diversity of possible histories. One of our variants builds less than 2 % of its final dynamical mass before reionization, rapidly quenching in-situ star formation. The bulk of its final stellar mass is later grown by dry mergers, depositing stars in the galaxys outskirts and hence expanding its effective radius. This mechanism constitutes a new formation scenario for highly diffuse ($text{r}_{1 /2} sim 820 , text{pc}$, $sim 32 , text{mag arcsec}^2$), metal-poor ($big[ mathrm{Fe}, / mathrm{H} big]= -2.9$), ultra-faint ($mathcal{M}_V= -5.7$) dwarf galaxies within the reach of next-generation low surface brightness surveys.