Continuum models for time-reversal (TR) invariant topological insulators (TIs) in $d geq 3$ dimensions are provided by harmonic oscillators coupled to certain $SO(d)$ gauge fields. These models are equivalent to the presence of spin-orbit (SO) interaction in the oscillator Hamiltonians at a critical coupling strength (equivalent to the harmonic oscillator frequency) and leads to flat Landau Level (LL) spectra and therefore to infinite degeneracy of either the positive or the negative helicity states depending on the sign of the SO coupling. Generalizing the results of Haaker et al. to $d geq 4$, we construct vector operators commuting with these Hamiltonians and show that $SO(d,2)$ emerges as the non-compact extended dynamical symmetry. Focusing on the model in four dimensions, we demonstrate that the infinite degeneracy of the flat spectra can be fully explained in terms of the discrete unitary representations of $SO(4,2)$, i.e. the {it doubletons}. The degeneracy in the opposite helicity branch is finite, but can still be explained exploiting the complex conjugate {it doubleton} representations. Subsequently, the analysis is generalized to $d$ dimensions, distinguishing the cases of odd and even $d$. We also determine the spectrum generating algebra in these models and briefly comment on the algebraic organization of the LL states w.r.t to an underlying deformed AdS geometry as well as on the organization of the surface states under open boundary conditions in view of our results.