We introduce a non-Abelian kagome lattice model that has both time-reversal and inversion symmetries and study the flat band physics and topological phases of this model. Due to the coexistence of both time-reversal and inversion symmetries, the energy bands consist of three doubly degenerate bands whose energy and conditions for the presence of flat bands could be obtained analytically, allowing us to tune the flat band with respect to the other two dispersive bands from the top to the middle and then to the bottom of the three bands. We further study the gapped phases of the model and show that they belong to the same phase as the band gaps only close at discrete points of the parameter space, making any two gapped phases adiabatically connected to each other without closing the band gap. Using the Pfaffian approach based on the time-reversal symmetry and parity characterization from the inversion symmetry, we calculate the bulk topological invariants and demonstrate that the unique gapped phases belong to the $Z_2$ quantum spin Hall phase, which is further confirmed by the edge state calculations.