Although Josephson junction qubits show great promise for quantum computing, the origin of dominant decoherence mechanisms remains unknown. We report Rabi oscillations for an improved phase qubit, and show that their coherence amplitude is significantly degraded by spurious microwave resonators. These resonators arise from changes in the junction critical current produced by two-level states in the tunnel barrier. The discovery of these high frequency resonators impacts the future of all Josephson qubits as well as existing Josephson technologies. We predict that removing or reducing these resonators through materials research will improve the coherence of all Josephson qubits.