We report the direct observation of slow fluctuations of helical antiferromagnetic domains in an ultra-thin holmium film using coherent resonant magnetic x-ray scattering. We observe a gradual increase of the fluctuations in the speckle pattern with increasing temperature, while at the same time a static contribution to the speckle pattern remains. This finding indicates that domain-wall fluctuations occur over a large range of time scales. We ascribe this non-ergodic behavior to the strong dependence of the fluctuation rate on the local thickness of the film.