Recent experiments have uncovered evidence of low energy excitations in the bulk of SmB$_6$ that are perhaps associated with unconventional quasiparticles, bringing into question whether this Kondo insulator is truly insulating in the bulk. Recently, we demonstrated that SmB$_6$ possesses significant in-gap bulk ac conduction far in excess of typical disordered semiconductors. Whether such conduction is an intrinsic feature of SmB$_6$, suggesting the formation of an exotic state, or residual conduction from impurities continues to be a topic of debate. Here, we further examine the origin of the ac optical conductivity of SmB$_6$ in light of recent experimental and theoretical developments. The optical conductivity of SmB$_6$ is shown to possess distinct regimes of either dominant free carrier or $localized$ response contributions. The free carrier response is found to be in good qualitative agreement with previous literature, although quantitative differences are revealed and discussed. The localized response, which dominates at the lowest temperatures, is analyzed in the context of models of either in-gap impurity states or an exotic neutral Fermi surface. The charge density or effective mass of this low temperature in-gap conductivity is extracted through a conductivity sum rule analysis and found to be in general alignment with both models in the appropriate limits. Our results shed further light on the nature of the in-gap states of this remarkable material.