We propose a new approach for high-contrast imaging at the diffraction limit using segmented telescopes in a modest observation bandwidth. This concept, named spectroscopic fourth-order coronagraphy, is based on a fourth-order coronagraph with a focal-plane mask that modulates the complex amplitude of the Airy disk along one direction. While coronagraphs applying the complex amplitude mask can achieve the theoretical limit performance for any arbitrary pupils, the focal plane mask severely limits the bandwidth. Here, focusing on the fact that the focal-plane mask modulates the complex amplitude along one direction, we noticed that the mask can be optimized for each spectral element generated by a spectrograph. We combine the fourth-order coronagraph with two spectrographs to produce a stellar spectrum on the focal plane and reconstruct a white pupil on the Lyot stop. Based on the wavefront analysis of an optical design applying an Offner-type imaging spectrograph, we found that the achievable contrast of this concept is 10^{-10} at 1.2 - 1.5 times the diffraction limit over the wavelength range of 650 - 750 nm for the entrance pupil of the LUVOIR telescope. Thus, this coronagraph concept could bring new habitable planet candidates not only around G- and K-type stars beyond 20 - 30 pc but also around very nearby M-type stars. This approach potentially promotes the characterization of the atmospheres of nearby terrestrial planets with future on- and off-axis segmented large telescopes.