Ultra-deep radio surveys are an invaluable probe of dust-obscured star formation, but require a clear understanding of the relative contribution from radio AGN to be used to their fullest potential. We study the composition of the $mu$Jy radio population detected in the Karl G. Jansky Very Large Array COSMOS-XS survey based on a sample of 1540 sources detected at 3 GHz over an area of $sim350text{arcmin}^2$. This ultra-deep survey consists of a single pointing in the well-studied COSMOS field at both 3 and 10 GHz and reaches RMS-sensitivities of $0.53$ and $0.41mu$Jy beam$^{-1}$, respectively. We find multi-wavelength counterparts for $97%$ of radio sources, based on a combination of near-UV/optical to sub-mm data, and through a stacking analysis at optical/near-infrared wavelengths we further show that the sources lacking such counterparts are likely to be high-redshift in nature (typical $zsim4-5$). Utilizing the multi-wavelength data over COSMOS, we identify AGN through a variety of diagnostics and find these to make up $23.2pm1.3%$ of our sample, with the remainder constituting uncontaminated star-forming galaxies. However, more than half of the AGN exhibit radio emission consistent with originating from star-formation, with only $8.8pm0.8%$ of radio sources showing a clear excess in radio luminosity. At flux densities of $sim30mu$Jy at 3 GHz, the fraction of star-formation powered sources reaches $sim90%$, and this fraction is consistent with unity at even lower flux densities. Overall, our findings imply that ultra-deep radio surveys such as COSMOS-XS constitute a highly effective means of obtaining clean samples of star-formation powered radio sources.