The random matrix theory (RMT) can be used to classify both topological phases of matter and quantum chaos. We develop a systematic and transformative RMT to classify the quantum chaos in the colored Sachdev-Ye-Kitaev (SYK) model first introduced by Gross and Rosenhaus. Here we focus on the 2-colored case and 4-colored case with balanced number of Majorana fermion $N$. By identifying the maximal symmetries, the independent parity conservation sectors, the minimum (irreducible) Hilbert space, and especially the relevant anti-unitary and unitary operators, we show that the color degree of freedoms lead to novel quantum chaotic behaviours. When $N$ is odd, different symmetry operators need to be constructed to make the classifications complete. The 2-colored case only show 3-fold Wigner-Dyson way, and the 4-colored case show 10-fold generalized Wigner-Dyson way which may also have non-trivial edge exponents. We also study 2- and 4-colored hybrid SYK models which display many salient quantum chaotic features hidden in the corresponding pure SYK models. These features motivate us to develop a systematic RMT to study the energy level statistics of 2 or 4 un-correlated random matrix ensembles. The exact diagonalizations are performed to study both the bulk energy level statistics and the edge exponents and find excellent agreements with our exact maximal symmetry classifications. Our complete and systematic methods can be easily extended to study the generic imbalanced cases. They may be transferred to the classifications of colored tensor models, quantum chromodynamics with pairings across different colors, quantum black holes and interacting symmetry protected (or enriched) topological phases.