Low-frequency $1/f^{gamma}$ noise is ubiquitous, even in high-end electronic devices. For qubits such noise results in decrease of their coherence times. Recently, it was found that adsorbed O$_2$ molecules provide the dominant contribution to flux noise in superconducting quantum interference devices. To clarify the basic principles of such adsorbant noise, we have investigated the formation of low-frequency noise while the mobility of surface adsorbants is varied by temperature. In our experiments, we measured low-frequency current noise in suspended monolayer graphene samples under the influence of adsorbed Ne atoms. Owing to the extremely small intrinsic noise of graphene in suspended Corbino geometry, we could resolve a combination of $1/f^{gamma}$ and Lorentzian noise spectra induced by the presence of Ne. We find that the $1/f^{gamma}$ noise is caused by surface diffusion of Ne atoms and by temporary formation of few-Ne-atom clusters. Our results support the idea that clustering dynamics of defects is relevant for understanding of $1/f$ noise in general metallic systems.