Monolayer FeSe on SrTiO$_3$ superconducts with reported $T_mathrm{c}$ as high as 100 K, but the dramatic interfacial $T_mathrm{c}$ enhancement remains poorly understood. Oxygen vacancies in SrTiO$_3$ are known to enhance the interfacial electron doping, electron-phonon coupling, and superconducting gap, but the detailed mechanism is unclear. Here we apply scanning transmission electron microscopy (STEM) and electron energy loss spectroscopy (EELS) to FeSe/SrTiO$_3$ to image the diffusion of selenium into SrTiO$_3$ to an unexpected depth of several unit cells, consistent with the simultaneously observed depth profile of oxygen vacancies. Our density functional theory (DFT) calculations support the crucial role of oxygen vacancies in facilitating the thermally driven Se diffusion. In contrast to excess Se in the FeSe monolayer or FeSe/SrTiO$_3$ interface that is typically removed during post-growth annealing, the diffused Se remains in the top few unit cells of the SrTiO$_3$ bulk after the extended post-growth annealing that is necessary to achieve superconductivity. Thus the unexpected Se in SrTiO$_3$ may contribute to the interfacial electron doping and electron-phonon coupling that enhance $T_mathrm{c}$, suggesting another important role for oxygen vacancies as facilitators of Se diffusion.