Helical Majorana fermions in d_{x^2-y^2} + i d_{xy}-wave topological superconductivity of doped correlated quantum spin Hall insulators

Large Hubbard U limit of the Kane-Mele model on a zigzag ribbon of honeycomb lattice near half-filling is studied via a renormalized mean-field theory. The ground state exhibits time-reversal symmetry (TRS) breaking d + i d-wave superconductivity. Atlarge spin-orbit coupling, the Z2 phase with non-trivial spin Chern number in the pure Kane-Mele model is persistent into the TRS broken state (called spin-Chern phase), and has two pairs of counter-propagating helical Majorana modes at the edges. As the spin-orbit coupling is reduced, the system undergoes a topological quantum phase transition from the spin-Chern to chiral superconducting states. Possible relevance of our results to adatom-doped graphene and irridate compounds is discussed.