Ginzburg-Landau vortices in superconductors attract or repel depending on whether the value of the coupling constant is less than 1 or larger than 1. At critical coupling it was previously observed that a strongly localised magnetic impurity behaves very similarly to a vortex. This remains true for axially symmetric configurations away from critical coupling. In particular, a delta function impurity of a suitable strength is related to a vortex configuration without impurity by singular gauge transformation. However, the interaction of vortices and impurities is more subtle and depends not only on the coupling constant and the impurity strength, but also on how broad the impurity is. Furthermore, the interaction typically depends on the distance and may be attractive at short distances and repulsive at long distances. Numerical simulations confirm moduli space approximation results for the scattering of one and two vortices with an impurity. However, a double vortex will split up when scattering with an impurity, and the direction of the split depends on the sign of the impurity. Head-on collisions of a single vortex with different impurities away from critical coupling is also briefly discussed.