The $N$-dimensional Smorodinsky-Winternitz system is a maximally superintegrable and exactly solvable model, being subject of study from different approaches. The model has been demonstrated to be multiseparable with wavefunctions given by Laguerre and Jacobi polynomials. In this paper we present the complete symmetry algebra ${cal SW}(N)$ of the system, which it is a higher-rank quadratic one containing the recently discovered Racah algebra ${cal R}(N)$ as subalgebra. The substructures of distinct quadratic ${cal Q}(3)$ algebras and their related Casimirs are also studied. In this way, from the constraints on the oscillator realizations of these substructures, the energy spectrum of the $N$-dimensional Smorodinsky-Winternitz system is obtained. We show that ${cal SW}(N)$ allows different set of substructures based on the Racah algebra ${cal R}({ N})$ which can be applied independently to algebraically derive the spectrum of the system.