Nowadays it is experimentally feasible to create artificial, and in particular, non-Abelian gauge potentials for ultracold atoms trapped in optical lattices. Motivated by this fact, we investigate the fundamental properties of an ultracold Fermi gas in a non-Abelian U(2) gauge potential characterized by a emph{constant} Wilson loop. Under this specific condition, the energy spectrum exhibits a robust band structure with large gaps and reveals a new fractal figure. The transverse conductivity is related to topological invariants and is shown to be quantized when the Fermi energy lies inside a gap of the spectrum. We demonstrate that the analogue of the integer quantum Hall effect for neutral atoms survives the non-Abelian coupling and leads to a striking fractal phase diagram. Moreover, this coupling induces an anomalous Hall effect as observed in graphene.