We present a novel approach to photometric redshifts, one that merges the advantages of both the template fitting and empirical fitting algorithms, without any of their disadvantages. This technique derives a set of templates, describing the spectral energy distributions of galaxies, from a catalog with both multicolor photometry and spectroscopic redshifts. The algorithm is essentially using the shapes of the templates as the fitting parameters. From simulated multicolor data we show that for a small training set of galaxies we can reconstruct robustly the underlying spectral energy distributions even in the presence of substantial errors in the photometric observations. We apply these techniques to the multicolor and spectroscopic observations of the Hubble Deep Field building a set of template spectra that reproduced the observed galaxy colors to better than 10%. Finally we demonstrate that these improved spectral energy distributions lead to a photometric-redshift relation for the Hubble Deep Field that is more accurate than standard template-based approaches.