The high room temperature mobility and the electron effective mass in BaSnO$_3$ are investigated in depth by evaluation of the free carrier absorption observed in infrared spectra for epitaxial films with free electron concentrations from $8.3 times 10^{18}$ to $7.3 times 10^{20}$~cm$^{-3}$. Both the optical band gap widening by conduction band filling and the carrier scattering mechanisms in the low and high doping regimes are consistently described employing parameters solely based on the intrinsic physical properties of BaSnO$_3$. The results explain the current mobility limits in epitaxial films and demonstrate the potential of BaSnO$_3$ to outperform established wide band gap semiconductors also in the moderate doping regime.