Noise of stochastic processes whose power spectrum scales at low frequencies, $f$, as $1/f$ appears in such diverse systems that it is considered universal. However, there have been a small number of instances from completely unrelated fields, e.g., the fluctuations of the human heartbeat or vortices in superconductors, in which power spectra have been observed to cross over from a $1/f$ to a non-$1/f$ behavior at even lower frequencies. Here, we show that such crossover must be universal, and can be accounted for by the memory of initial conditions and the relaxation processes present in any physical system. When the smallest frequency allowed by the experimental observation time, $omega_{obs}$, is larger than the smallest relaxation frequency, $Omega_{min}$, a $1/f$ power spectral density is obtained. Conversely, when $omega_{obs}<Omega_{min}$ we predict that the power spectrum of any stochastic process should exhibit a crossover from $1/f$ to a different, integrable functional form provided there is enough time for experimental observations. This crossover also provides a convenient tool to measure the lowest relaxation frequency of a physical system.