The abundances of r-process elements of very metal-poor stars capture the history of the r-process enrichment in the early stage of star formation in a galaxy. Currently, various types of astrophysical sites including neutron star mergers, magneto-rotational supernovae, and collapsars, are suggested as the origin of r-process elements. The time delay between the star formation and the production of r-process elements is the key to distinguish these scenarios with the caveat that the diffusion of r-process elements in the interstellar medium may induce the delay in r-process enrichment because r-process events are rare. Here we study the observed Ba abundance data of very metal-poor stars as the tracer of the early enrichment history of r-process elements. We find that the gradual increase of [Ba/Mg] with [Fe/H], which is remarkably similar among the Milky Way and classical dwarfs, requires a significant time delay (100 Myr -- 1 Gyr) of r-process events from star formation rather than the diffusion-induced delay. We stress that this conclusion is robust to the assumption regarding s-process contamination in the Ba abundances because the sources with no delay would overproduce Ba at very low metallicities even without the contribution from the s-process. Therefore we conclude that sources with a delay, possibly neutron star mergers, are the origins of r-process elements.