The rotational period of isolated pulsars increases over time due to the extraction of angular momentum by electromagnetic torques. These torques also change the obliquity angle $alpha$ between the magnetic and rotational axes. Although actual pulsar magnetospheres are plasma-filled, the time evolution of $alpha$ has mostly been studied for vacuum pulsar magnetospheres. In this work, we self-consistently account for the plasma effects for the first time by analysing the results of time-dependent 3D force-free and magnetohydrodynamic simulations of pulsar magnetospheres. We show that if a neutron star is spherically symmetric and is embedded with a dipolar magnetic moment, the pulsar evolves so as to minimise its spin-down luminosity: both vacuum and plasma-filled pulsars evolve toward the aligned configuration ($alpha=0$). However, they approach the alignment in qualitatively different ways. Vacuum pulsars come into alignment exponentially fast, with $alpha propto exp(-t/tau)$ and $tau sim$ spindown timescale. In contrast, we find that plasma-filled pulsars align much more slowly, with $alpha propto (t/tau)^{-1/2}$. We argue that the slow time evolution of obliquity of plasma-filled pulsars can potentially resolve several observational puzzles, including the origin of normal pulsars with periods of $sim1$ second, the evidence that oblique pulsars come into alignment over a timescale of $sim 10^7$ years, and the observed deficit, relative to an isotropic obliquity distribution, of pulsars showing interpulse emission.