We present measurements of 1/f frequency noise in both linear and Josephson-junction-embedded superconducting aluminum resonators in the low power, low temperature regime - typical operating conditions for superconducting qubits. The addition of the Josephson junction does not result in additional frequency noise, thereby placing an upper limit for fractional critical current fluctuations of $10^{-8}$ (Hz$^{-1/2}$) at 1 Hz for sub-micron, shadow evaporated junctions. These values imply a minimum dephasing time for a superconducting qubit due to critical current noise of 40 -- 1400 $mu$s depending on qubit architecture. Occasionally, at temperatures above 50 mK, we observe the activation of individual fluctuators which increase the level of noise significantly and exhibit Lorentzian spectra.