Unwanted fluctuations over time, in short, noise, are detrimental to device performance, especially for quantum coherent circuits. Recent efforts have demonstrated routes to utilizing magnon systems for quantum technologies, which are based on interfacing single magnons to superconducting qubits. However, the coupling of several components often introduces additional noise to the system, degrading its coherence. Researching the temporal behavior can help to identify the underlying noise sources, which is a vital step in increasing coherence times and the hybrid device performance. Yet, the frequency noise of the ferromagnetic resonance (FMR) has so far been unexplored. Here, we investigate such FMR frequency fluctuations of a YIG sphere down to mK-temperatures, and find them independent of temperature and drive power. This suggests that the measured frequency noise in YIG is dominated by so far undetermined noise sources, which properties are not consistent with the conventional model of two-level systems, despite their effect on the sample linewidth. Moreover, the functional form of the FMR frequency noise power spectral density (PSD) cannot be described by a simple power law. By employing time-series analysis, we find a closed function for the PSD that fits our observations. Our results underline the necessity of coherence improvements to magnon systems for useful applications in quantum magnonics.