We expand the concept of frustration in Mott insulators and quantum spin liquids to metals with flat bands. We show that when inter-orbital hopping $t_2$ dominates over intra-orbital hopping $t_1$, in a multiband system with strong spin-orbit coupling $lambda$, electronic states with a narrow bandwidth $Wsim t_2^2/lambda$ are formed compared to a bandwidth of order $t_1$ for intra-orbital hopping. We demonstrate the evolution of the electronic structure, Berry phase distributions for time-reversal and inversion breaking cases, and their imprint on the optical absorption, in a tight binding model of $d$-orbital hopping on a honeycomb lattice. Going beyond quantum Hall effect and twisted bilayer graphene, we provide an alternative mechanism and a richer materials platform for achieving flat bands poised at the brink of instabilities toward novel correlated and fractionalized metallic phases.