A precise measurement of dephasing over a range of timescales is critical for improving quantum gates beyond the error correction threshold. We present a metrological tool, based on randomized benchmarking, capable of greatly increasing the precision of Ramsey and spin echo sequences by the repeated but incoherent addition of phase noise. We find our SQUID-based qubit is not limited by $1/f$ flux noise at short timescales, but instead observe a telegraph noise mechanism that is not amenable to study with standard measurement techniques.