We have analyzed high resolution N-body simulations of dark matter halos, focusing specifically on the evolution of angular momentum. We find that not only is individual particle angular momentum not conserved, but the angular momentum of radial shells also varies over the age of the Universe by up to factors of a few. We find that torques from external structure are the most likely cause for this distribution shift. Since the model of adiabatic contraction that is often applied to model the effects of galaxy evolution on the dark-matter density profile in a halo assumes angular momentum conservation, this variation implies that there is a fundamental limit on the possible accuracy of the adiabatic contraction model in modeling the response of DM halos to the growth of galaxies.