We report on the observation of a large anisotropy in the rethermalization dynamics of an ultracold dipolar Fermi gas driven out of equilibrium. Our system consists of an ultracold sample of strongly magnetic $^{167}$Er fermions, spin-polarized in the lowest Zeeman sublevel. In this system, elastic collisions arise purely from universal dipolar scattering. Based on cross-dimensional rethermalization experiments, we observe a strong anisotropy of the scattering, which manifests itself in a large angular dependence of the thermal relaxation dynamics. Our result is in very good agreement with recent theoretical predictions. Furthermore, we measure the rethermalization rate as a function of temperature for different angles and find that the suppression of collisions by Pauli blocking is not influenced by the dipole orientation.