We systematically investigate the influence of high pressure on the electronic transport properties of layered ferromagnetic materials, in particular, those of Fe$_3$GeTe$_2$. Its crystal sustains a hexagonal phase under high pressures up to 25.9 GPa, while the Curie temperature decreases monotonously with the increasing pressure. By applying appropriate pressures, the experimentally measured anomalous Hall conductivity, $sigma_{xy}^A$, can be efficiently controlled. Our theoretical study reveals that this finding can be attributed to the shift of the spin--orbit-coupling-induced splitting bands of Fe atoms. With loading compression, $sigma_{xy}^A$ reaches its maximal value when the Fermi level lies inside the splitting bands and then attenuates when the splitting bands float above the Fermi level. Further compression leads to a prominent suppression of the magnetic moment, which is another physical cause of the decrease in $sigma_{xy}^A$ at high pressure. These results indicate that the application of pressure is an effective approach in controlling the anomalous Hall conductivity of layered magnetic materials, which elucidates the physical mechanism of the large intrinsic anomalous Hall effect.