Ferromagnetic quantum critical points were predicted to be prohibited in clean itinerant ferromagnetic systems, yet such a phenomenon was recently revealed in CeRh$_6$Ge$_4$, where the Curie temperature can be continuously suppressed to zero under a moderate hydrostatic pressure. Here we report the observation of quantum oscillations in CeRh$_6$Ge$_4$ from measurements using the cantilever and tunnel-diode oscillator methods in fields up to 45 T, clearly demonstrating that the ferromagnetic quantum criticality occurs in a clean system. In order to map the Fermi surface of CeRh$_6$Ge$_4$, we performed angle-dependent measurements of quantum oscillations at ambient pressure, and compared the results to density functional theory calculations. The results are consistent with the Ce 4f electrons remaining localized, and not contributing to the Fermi surface, suggesting that localized ferromagnetism is a key factor for the occurrence of a ferromagnetic quantum critical point in CeRh$_6$Ge$_4$.