The impact of non-magnetic and magnetic impurities on topological insulators is a central problem concerning their fundamental physics and possible novel spintronics and quantum computing applications. SmB$_6$, predicted to be a topological Kondo insulator, is considered a benchmark material. Using a spin-polarized tip in scanning tunneling spectroscopy destroys the signature peak of the topological surface state, revealing its spin texture. Further, combining local STS with macroscopic transport measurements on SmB$_6$ containing different substitutions enables us to investigate the effect of impurities. The surface states around impurities are locally suppressed with different length scales depending on their magnetic properties and, for sufficiently high impurity level, globally destroyed. Our study points directly to the topological nature of SmB$_6$, and unveils, microscopically and macroscopically, how impurities -- magnetic or non-magnetic -- affect topological surface states.