We prove that the perverse Leray filtration for the Hitchin morphism is locally constant in families, thus providing some evidence towards the validity of the $P=W$ conjecture due to de Cataldo, Hausel and Migliorini in non Abelian Hodge theory.