We investigate the dynamics of a quantized vortex and a nuclear impurity immersed in a neutron superfluid within a fully microscopic time-dependent three-dimensional approach. The magnitude and even the sign of the force between the quantized vortex and the nuclear impurity have been a matter of debate for over four decades. We determine that the vortex and the impurity repel at neutron densities, 0.014 fm$^{-3}$ and 0.031 fm$^{-3}$, which are relevant to the neutron star crust and the origin of glitches, while previous calculations have concluded that the force changes its sign between these two densities and predicted contradictory signs. The magnitude of the force increases with the density of neutron superfluid, while the magnitude of the pairing gap decreases in this density range.