Infrared-faint radio sources (IFRS) are objects that have flux densities of several mJy at 1.4GHz, but that are invisible at 3.6um when using sensitive Spitzer observations with uJy sensitivities. Their nature is unclear and difficult to investigate since they are only visible in the radio. High-resolution radio images and comprehensive spectral coverage can yield constraints on the emission mechanisms of IFRS and can give hints to similarities with known objects. We imaged a sample of 17 IFRS at 4.8GHz and 8.6GHz with the Australia Telescope Compact Array to determine the structures on arcsecond scales. We added radio data from other observing projects and from the literature to obtain broad-band radio spectra. We find that the sources in our sample are either resolved out at the higher frequencies or are compact at resolutions of a few arcsec, which implies that they are smaller than a typical galaxy. The spectra of IFRS are remarkably steep, with a median spectral index of -1.4 and a prominent lack of spectral indices larger than -0.7. We also find that, given the IR non-detections, the ratio of 1.4GHz flux density to 3.6um flux density is very high, and this puts them into the same regime as high-redshift radio galaxies. The evidence that IFRS are predominantly high-redshift sources driven by active galactic nuclei (AGN) is strong, even though not all IFRS may be caused by the same phenomenon. Compared to the rare and painstakingly collected high-redshift radio galaxies, IFRS appear to be much more abundant, but less luminous, AGN-driven galaxies at similar cosmological distances.