$beta$-Ga$_2$O$_3$ is an ultra-wide bandgap semiconductor and is thus expected to be optically transparent to light of sub-bandgap wavelengths well into the ultraviolet. Contrary to this expectation, it is found here that free electrons in n-doped $beta$-Ga$_2$O$_3$ absorb light from the IR to the UV wavelength range via intra- and inter-conduction band optical transitions. Intra-conduction band absorption occurs via an indirect optical phonon mediated process with a $1/omega^{3}$ dependence in the visible to near-IR wavelength range. This frequency dependence markedly differs from the $1/omega^{2}$ dependence predicted by the Drude model of free-carrier absorption. The inter-conduction band absorption between the lowest conduction band and a higher conduction band occurs via a direct optical process at $lambda sim 349$ nm (3.55 eV). Steady state and ultrafast optical spectroscopy measurements unambiguously identify both these absorption processes and enable quantitative measurements of the inter-conduction band energy, and the frequency dependence of absorption. Whereas the intra-conduction band absorption does not depend on light polarization, inter-conduction band absorption is found to be strongly polarization dependent. The experimental observations, in excellent agreement with recent theoretical predictions for $beta$-Ga$_2$O$_3$, provide important limits of sub-bandgap transparency for optoelectronics in the deep-UV to visible wavelength range, and are also of importance for high electric field transport effects in this emerging semiconductor.