We determine the intrinsic, 3-dimensional shape distribution of star-forming galaxies at 0<z<2.5, as inferred from their observed projected axis ratios. In the present-day universe star-forming galaxies of all masses 1e9 - 1e11 Msol are predominantly thin, nearly oblate disks, in line with previous studies. We now extend this to higher redshifts, and find that among massive galaxies (M* > 1e10 Msol) disks are the most common geometric shape at all z < 2. Lower-mass galaxies at z>1 possess a broad range of geometric shapes: the fraction of elongated (prolate) galaxies increases toward higher redshifts and lower masses. Galaxies with stellar mass 1e9 Msol (1e10 Msol) are a mix of roughly equal numbers of elongated and disk galaxies at z~1 (z~2). This suggests that galaxies in this mass range do not yet have disks that are sustained over many orbital periods, implying that galaxies with present-day stellar mass comparable to that of the Milky Way typically first formed such sustained stellar disks at redshift z~1.5-2. Combined with constraints on the evolution of the star formation rate density and the distribution of star formation over galaxies with different masses, our findings imply that, averaged over cosmic time, the majority of stars formed in disks.