Long-lived, high-frequency phonons are valuable for applications ranging from optomechanics to emerging quantum systems. For scientific as well as technological impact, we seek high-performance oscillators that offer a path towards chip-scale integration. Confocal bulk acoustic wave resonators have demonstrated an immense potential to support long-lived phonon modes in crystalline media at cryogenic temperatures. So far, these devices have been macroscopic with cm-scale dimensions. However, as we push these oscillators to high frequencies, we have an opportunity to radically reduce the footprint as a basis for classical and emerging quantum technologies. In this paper, we present novel design principles and simple fabrication techniques to create high performance chip-scale confocal bulk acoustic wave resonators in a wide array of crystalline materials. We tailor the acoustic modes of such resonators to efficiently couple to light, permitting us to perform a non-invasive laser-based phonon spectroscopy. Using this technique, we demonstrate an acoustic $Q$-factor of 28 million (6.5 million) for chip-scale resonators operating at 12.7 GHz (37.8 GHz) in crystalline $z$-cut quartz ($x$-cut silicon) at cryogenic temperatures.