Dilute magnetic semiconductors (DMSs) show great promise for applications in spin-based electronics, but in most cases continue to elude explanations of their magnetic behavior. Here, we combine quantitative x-ray spectroscopy and Anderson impurity model calculations to study ferromagnetic Fe-substituted In$_2$O$_3$ films, and we identify a subset of Fe atoms adjacent to oxygen vacancies in the crystal lattice which are responsible for the observed room temperature ferromagnetism. Using resonant inelastic x-ray scattering, we map out the near gap electronic structure and provide further support for this conclusion. Serving as a concrete verification of recent theoretical results and indirect experimental evidence, these results solidify the role of impurity-vacancy coupling in oxide-based DMSs.