Ultrafast processes in matter, such as the electron emission following light absorption, can now be studied using ultrashort light pulses of attosecond duration ($10^{-18}$s) in the extreme ultraviolet spectral range. The lack of spectral resolution due to the use of short light pulses may raise serious issues in the interpretation of the experimental results and the comparison with detailed theoretical calculations. Here, we determine photoionization time delays in neon atoms over a 40 eV energy range with an interferometric technique combining high temporal and spectral resolution. We spectrally disentangle direct ionization from ionization with shake up, where a second electron is left in an excited state, thus obtaining excellent agreement with theoretical calculations and thereby solving a puzzle raised by seven-year-old measurements. Our experimental approach does not have conceptual limits, allowing us to foresee, with the help of upcoming laser technology, ultra-high resolution time-frequency studies from the visible to the x-ray range.