The low energy magnetic excitation spectrum of the Heisenberg antiferromagnetic $S = 1/2$ chain system Sr$_2$CuO$_3$ with Ni- and Ca-impurities is studied by neutron spectroscopy. In all cases, a defect-induced spectral pseudogap is observed and shown to scale proportionately to the number of scattering centers in the spin chains.