Resistivity, magnetization and microscopic $^{75}$As nuclear magnetic resonance (NMR) measurements in the antiferromagnetically ordered state of the iron-based superconductor parent material CaFe$_2$As$_2$ exhibit anomalous features that are consistent with the collective freezing of domain walls. Below $T^*approx 10$ K, the resistivity exhibits a peak and downturn, the bulk magnetization exhibits a sharp increase, and $^{75}$As NMR measurements reveal the presence of slow fluctuations of the hyperfine field. These features in both the charge and spin response are strongly field dependent, are fully suppressed by $H^*approx 15$ T, and suggest the presence of filamentary superconductivity nucleated at the antiphase domain walls in this material.