CrAs is a well-known helimagnet with the double-helix structure originating from the competition between the Dzyaloshinskii-Moriya interaction (DMI) and antiferromagnetic exchange interaction $J$. By resonant soft X-ray scattering (RSXS), we observe the magnetic peak (0~0~$q_m$) that emerges at the helical transition with $T_S$ $approx$ 267.5 K. Intriguingly, the helimagnetic domains significantly shrink on cooling below $sim$255 K, opposite to the conventional thermal effect. The weakening of DMI on cooling is found to play a critical role here. It causes the helical wave vector to vary, ordered spins to rotate, and extra helimagnetic domain boundaries to form at local defects, thus leading to the anomalous shrinkage of helimagnetic domains. Our results indicate that the size of magnetic helical domains can be controlled by tuning DMI in certain helimagnets.