We experimentally investigate spin-orbit torque and spin pumping in Y$_3$Fe$_5$O$_{12}$(YIG)/Pt bilayers with ultrathin insertion layers at the interface. An insertion layer of Cu suppresses both spin-orbit torque and spin pumping, whereas an insertion layer of Ni$_{80}$Fe$_{20}$ (permalloy, Py) enhances them, in a quantitatively consistent manner with the reciprocity of the two spin transmission processes. However, we observe a large enhancement of Gilbert damping with the insertion of Py that cannot be accounted for solely by spin pumping, suggesting significant spin-memory loss due to the interfacial magnetic layer. Our findings indicate that the magnetization at the YIG-metal interface strongly influences the transmission and depolarization of pure spin current.