One of the main goals of modern observational cosmology is to map the large scale structure of the Universe. A potentially powerful approach for doing this would be to exploit three-dimensional spectral maps, i.e. the specific intensity of extragalactic light as a function of wavelength and direction on the sky, to measure spatial variations in the total extragalactic light emission and use these as a tracer of the clustering of matter. A main challenge is that the observed intensity as a function of wavelength is a convolution of the source luminosity density with the rest-frame spectral energy distribution. In this paper, we introduce the method of spectral deconvolution as a way to invert this convolution and extract the clustering information. We show how one can use observations of the mean and angular fluctuations of extragalactic light as a function of wavelength, assuming statistical isotropy, to reconstruct jointly the rest-frame spectral energy distribution of the sources and the source spatial density fluctuations. This method is more general than the well known line mapping technique as it does not rely on spectral lines in the emitted spectra. After introducing the general formalism, we discuss its implementation and limitations. This formal paper sets the stage for future more practical studies.