The properties of $Xi^-$ hypernuclei are studied systematically using a two-dimensional Skyrme-Hartree-Fock approach combined with three different $Xi N$ Skyrme forces fitted to reproduce the existing data. We explore the impurity effect of a single $Xi^-$ hyperon on the radii, deformations, and density distributions of the nuclear core and point out qualitative differences between the different forces. We find that the $Xi^-$ removal energy of $^{hskip0.10em13}_{Xi p}$B [$^{12}$C(g.s.)+ $Xi^-$(1p)] calculated by the SLX3 force is 0.7 MeV, which is in good agreement with a possible value of $0.82pm0.17;$MeV from the KEK E176 experiment. The theoretical prediction for this weakly bound state depends strongly on the deformation of the nuclear core, which is analyzed in detail.