An important challenge in the field of many-body quantum dynamics is to identify non-ergodic states of matter beyond many-body localization (MBL). Strongly disordered spin chains with non-Abelian symmetry and chains of non-Abelian anyons are natural candidates, as they are incompatible with standard MBL. In such chains, real space renormalization group methods predict a partially localized, non-ergodic regime known as a quantum critical glass (a critical variant of MBL). This regime features a tree-like hierarchy of integrals of motion and symmetric eigenstates with entanglement entropy that scales as a logarithmically enhanced area law. We argue that such tentative non-ergodic states are perturbatively unstable using an analytic computation of the scaling of off-diagonal matrix elements and accessible level spacing of local perturbations. Our results indicate that strongly disordered chains with non-Abelian symmetry display either spontaneous symmetry breaking or ergodic thermal behavior at long times. We identify the relevant length and time scales for thermalization: even if such chains eventually thermalize, they can exhibit non-ergodic dynamics up to parametrically long time scales with a non-analytic dependence on disorder strength.