During defect-antidefect scattering, bound modes frequently disappear into the continuous spectrum before the defects themselves collide. This leads to a structural, nonperturbative change in the spectrum of small excitations. Sometimes the effect can be seen as a hard wall from which the defect can bounce off. We show the existence of these spectral walls and study their properties in the $phi^4$ model with BPS preserving impurity, where the spectral wall phenomenon can be isolated because the static force between the antikink and the impurity vanishes. We conclude that such spectral walls should surround all solitons possessing internal modes.