We present an atlas of Spitzer/IRS high resolution (R~600) 10-37um spectra for 24 well known starburst galaxies. The spectra are dominated by fine-structure lines, molecular hydrogen lines, and emission bands of polycyclic aromatic hydrocarbons. Six out of the eight objects with a known AGN component show emission of the high excitation [NeV] line. This line is also seen in one other object (NGC4194) with, a priori, no known AGN component. In addition to strong polycyclic aromatic hydrocarbon emission features in this wavelength range (11.3, 12.7, 16.4um), the spectra reveal other weak hydrocarbon features at 10.6, 13.5, 14.2um, and a previously unreported emission feature at 10.75um. An unidentified absorption feature at 13.7um is detected in many of the starbursts. We use the fine-structure lines to derive the abundance of neon and sulfur for 14 objects where the HI 7-6 line is detected. We further use the molecular hydrogen lines to sample the properties of the warm molecular gas. Several basic diagrams characterizing the properties of the sample are also shown. We have combined the spectra of all the pure starburst objects to create a high S/N template, which is available to the community.