We observe the generation of high harmonics in the plane perpendicular to the driving laser polarization and show that these are driven by the spin-orbit interaction. Using R-Matrix with time-dependence theory, we demonstrate that for certain initial states either circularly- or linearly- polarized harmonics arise via well-known selection rules between atomic states controlled by the spin-orbit interaction. Finally, we elucidate the connection between the observed harmonics and the phase of the intial state.