We calculate the radio-frequency spectrum of balanced and imbalanced ultracold Fermi gases in the normal phase at unitarity. For the homogeneous case the spectrum of both the majority and minority components always has a single peak even in the pseudogap regime. We furthermore show how the double-peak structures observed in recent experiments arise due to the inhomogeneity of the trapped gas. The main experimental features observed above the critical temperature in the recent experiment of Schunck et al. [Science 316, 867, (2007)] are recovered with no fitting parameters.