Machine-Learned Phase Diagrams of Generalized Kitaev Honeycomb Magnets

We use a recently developed interpretable and unsupervised machine-learning method, the tensorial kernel support vector machine (TK-SVM), to investigate the low-temperature classical phase diagram of a generalized Heisenberg-Kitaev-$Gamma$ ($J$-$K$-$Gamma$) model on a honeycomb lattice. Aside from reproducing phases reported by previous quantum and classical studies, our machine finds a hitherto missed nested zigzag-stripy order and establishes the robustness of a recently identified modulated $S_3 times Z_3$ phase, which emerges through the competition between the Kitaev and $Gamma$ spin liquids, against Heisenberg interactions. The results imply that, in the restricted parameter space spanned by the three primary exchange interactions -- $J$, $K$, and $Gamma$, the representative Kitaev material $alpha$-${rm RuCl}_3$ lies close to the boundaries of several phases, including a simple ferromagnet, the unconventional $S_3 times Z_3$ and nested zigzag-stripy magnets. A zigzag order is stabilized by a finite $Gamma^{prime}$ and/or $J_3$ term, whereas the four magnetic orders may compete in particular if $Gamma^{prime}$ is anti-ferromagnetic.