More than forty years ago, Barash published a calculation of the full retarded Casimir-Lifshitz torque for planar birefringent media with arbitrary degrees of anisotropy. An independent theoretical confirmation has been lacking since. We report a systematic and transparent derivation of the torque between two media with both electric and magnetic birefringence. Our approach, based on an eigenmode decomposition of Maxwells equations, generalizes Barashs result for electrically birefringent materials, and can be generalized to a wide range of anisotropic materials and finite thickness effects.