Using the deepest 1.2 mm continuum map to date in the Hubble Ultra Deep Field obtained as part of the ALMA Spectroscopic Survey (ASPECS) large program, we measure the cosmic density of dust and implied gas (H$_{2}+$H I) mass in galaxies as a function of look-back time. We do so by stacking the contribution from all $H$-band selected galaxies above a given stellar mass in distinct redshift bins, $rho_{rm dust}(M_ast>M,z)$ and $rho_{rm gas}(M_ast>M,z)$. At all redshifts, $rho_{rm dust}(M_ast>M,z)$ and $rho_{rm gas}(M_ast>M,z)$ grow rapidly as $M$ decreases down to $10^{10},M_odot$, but this growth slows down towards lower stellar masses. This flattening implies that at our stellar mass-completeness limits ($10^8,M_odot$ and $10^{8.9},M_odot$ at $zsim0.4$ and $zsim3$), both quantities converge towards the total cosmic dust and gas mass densities in galaxies. The cosmic dust and gas mass densities increase at early cosmic time, peak around $zsim2$, and decrease by a factor $sim4$ and 7, compared to the density of dust and molecular gas in the local universe, respectively. The contribution of quiescent galaxies -- i.e., with little on-going star-formation -- to the cosmic dust and gas mass densities is minor ($lesssim10%$). The redshift evolution of the cosmic gas mass density resembles that of the star-formation rate density, as previously found by CO-based measurements. This confirms that galaxies have relatively constant star-formation efficiencies (within a factor $sim2$) across cosmic time. Our results also imply that by $zsim0$, a large fraction ($sim90%$) of dust formed in galaxies across cosmic time has been destroyed or ejected to the intergalactic medium.