It has been suggested based on analytic theory that even in non-rotating supernova progenitors stochastic spin-up by internal gravity waves (IGWs) during the late burning stages can impart enough angular momentum to the core to result in neutron star birth spin periods below 100ms, and a relatively firm upper limit of 500ms for the spin period. We here investigate this process using a 3D simulation of oxygen shell burning in a $3M_odot$ He star. Our model indicates that stochastic spin-up by IGWs is less efficient than previously thought. We find that the stochastic angular momentum flux carried by waves excited at the shell boundary is significantly smaller for a given convective luminosity and turnover time than would be expected from simple dimensional analysis. This can be explained by noting that the waves launched by overshooting convective plumes contain modes of opposite angular wave number with similar amplitudes, so that the net angular momentum of excited wave packets almost cancels. We find that the wave-mediated angular momentum flux from the oxygen shell follows a random walk, but again dimensional analysis overestimates the random walk amplitudes since the correlation time is only a fraction of the convective turnover time. Extrapolating our findings over the entire life time of the last burning stages prior to collapse, we predict that the core angular momentum from stochastic spin-up would translate into long birth spin periods of several seconds for low-mass progenitors and no less than 100ms even for high-mass progenitors.