The Berry curvature in magnetic systems is attracting interest due to the potential tunability of topological features via the magnetic structure. $f$-electrons, with their large spin-orbit coupling, abundance of non-collinear magnetic structures and high electronic tunability, are attractive candidates to search for tunable topological properties. In this study, we measure anomalous Hall effect (AHE) in the distorted kagom$acute{e}$ heavy fermion antiferromagnet U$_3$Ru$_4$Al$_{12}$. A large intrinsic AHE in high fields reveals the presence of a large Berry curvature. Moreover, the fields required to obtain the large Berry curvature are significantly different between $B parallel a$ and $B parallel a^*$, providing a mechanism to control the topological response in this system. Theoretical calculations illustrate that this sensitivity may be due to the heavy fermion character of the electronic structure. These results shed light on the Berry curvature of a strongly correlated band structure in magnetically frustrated heavy fermion materials, but also emphasize 5$f$-electrons as an ideal playground for studying field-tuned topological states.