The true topological nature of the Kondo insulator SmB$_6$ remains to be unveiled. Our previous tunneling study not only found evidence for the existence of surface Dirac fermions, but it also uncovered that they inherently interact with the spin excitons, collective excitations in the bulk. We have extended such a spectroscopic investigation into crystals containing a Sm deficiency. The bulk hybridization gap is found to be insensitive to the deficiency up to 1% studied here, but the surface states in Sm-deficient crystals exhibit quite different temperature evolutions from those in stoichiometric ones. We attribute this to the topological surface states remaining incoherent down to the lowest measurement temperature due to their continued interaction with the spin excitons that remain uncondensed. This result shows that the detailed topological nature of SmB$_6$ could vary drastically in the presence of disorder in the lattice. This sensitiveness to disorder is seemingly contradictory to the celebrated topological protection, but it can be understood as being due to the intimate interplay between strong correlations and topological effects.