We present a local probe study of the magnetic superconductor, ErNi$_2$B$_2$C, using magnetic force microscopy at sub-Kelvin temperatures. ErNi$_2$B$_2$C is an ideal system to explore the effects of concomitant superconductivity and ferromagnetism. At 500 mK, far below the transition to a weakly ferromagnetic state, we directly observe a structured magnetic background on the micrometer scale. We determine spatially resolved absolute values of the magnetic penetration depth $lambda$ and study its temperature dependence as the system undergoes magnetic phase transitions from paramagnetic to antiferromagnetic, and to weak ferromagnetic, all within the superconducting regime. In addition, we estimate the absolute pinning force of Abrikosov vortices, which shows a position- and temperature dependence as well, and discuss the possibility of the purported spontaneous vortex formation.