The Hilbert space of a quantum system with internal global symmetry $G$ decomposes into sectors labelled by irreducible representations of $G$. If the system is chaotic, the energies in each sector should separately resemble ordinary random matrix theory. We show that such sector-wise random matrix ensembles arise as the boundary dual of two-dimensional gravity with a $G$ gauge field in the bulk. Within each sector, the eigenvalue density is enhanced by a nontrivial factor of the dimension of the representation, and the ground state energy is determined by the quadratic Casimir. We study the consequences of t Hooft anomalies in the matrix ensembles, which are incorporated by adding specific topological terms to the gauge theory action. The effect is to introduce projective representations into the decomposition of the Hilbert space. Finally, we consider ensembles with $G$ symmetry and time reversal symmetry, and analyze a simple case of a mixed anomaly between time reversal and an internal $mathbb{Z}_2$ symmetry.