We show that the infinitesimal deformations of the Brill--Noether locus $W_d$ attached to a smooth non-hyperelliptic curve $C$ are in one-to-one correspondence with the deformations of $C$. As an application, we prove that if a Jacobian $J$ deforms together with a minimal cohomology class out the Jacobian locus, then $J$ is hyperelliptic. In particular, this provides an evidence to a conjecture of Debarre on the classification of ppavs carrying a minimal cohomology class. Finally, we also study simultaneous deformations of Fano surfaces of lines and intermediate Jacobians.