We have prepared high-quality epitaxial thin films of CaRuO$_3$ with residual resistivity ratios up to 55. Shubnikov-de Haas oscillations in the magnetoresistance and a $T^2$ temperature dependence in the electrical resistivity only below 1.5 K, whose coefficient is substantially suppressed in large magnetic fields, establish CaRuO$_3$ as a Fermi liquid (FL) with anomalously low coherence scale. Non-Fermi liquid (NFL) $T^{3/2}$ dependence is found between 2 and 25 K. The high sample quality allows access to the intrinsic electronic properties via THz spectroscopy. For frequencies below 0.6 THz, the conductivity is Drude-like and can be modeled by FL concepts, while for higher frequencies non-Drude behavior, inconsistent with FL predictions, is found. This establishes CaRuO$_3$ as a prime example of optical NFL behavior in the THz range.