We report time-resolved magneto-optic Kerr effect measurements of the longitudinal spin Seebeck effect driven by an interfacial temperature difference between itinerant electrons and magnons. The measured time-evolution of spin accumulation induced by laser-excitation indicates transfer of angular momentum across Au/Y$_3$Fe$_5$O$_{12}$ and Cu/Y$_3$Fe$_5$O$_{12}$ interfaces on a picosecond time-scale. The product of spin-mixing conductance and interfacial spin Seebeck coefficient determined is of the order of $10^8$ A m$^{-2}$ K$^{-1}$.