We resolve phonon number states in the spectrum of a superconducting qubit coupled to a multimode acoustic cavity. Crucial to this resolution is the sharp frequency dependence in the qubit-phonon interaction engineered by coupling the qubit to surface acoustic waves in two locations separated by $sim40$ acoustic wavelengths. In analogy to double-slit diffraction, the resulting self-interference generates high-contrast frequency structure in the qubit-phonon interaction. We observe this frequency structure both in the coupling rate to multiple cavity modes and in the qubit spontaneous emission rate into unconfined modes. We use this sharp frequency structure to resolve single phonons by tuning the qubit to a frequency of destructive interference where all acoustic interactions are dispersive. By exciting several detuned yet strongly-coupled phononic modes and measuring the resulting qubit spectrum, we observe that, for two modes, the device enters the strong dispersive regime where single phonons are spectrally resolved.