CeRh$_6$Ge$_4$ is an unusual example of a stoichiometric heavy fermion ferromagnet, which can be cleanly tuned by hydrostatic pressure to a quantum critical point. In order to understand the origin of this anomalous behavior, we have characterized the magnetic ordering and crystalline electric field (CEF) scheme of this system. While magnetic Bragg peaks are not resolved in neutron powder diffraction, coherent oscillations are observed in zero-field $mu$SR below $T_{rm C}$, which are consistent with in-plane ferromagnetic ordering consisting of reduced Ce moments. From analyzing the magnetic susceptibility and inelastic neutron scattering, we propose a CEF-level scheme which accounts for the easy-plane magnetocrystalline anisotropy, and suggests that the orbital anisotropy of the ground state and low lying excited state doublets is an important factor giving rise to the observed anisotropic hybridization.