Inferring the climate and surface conditions of terrestrial exoplanets in the habitable zone is a major goal for the field of exoplanet science. This pursuit will require both statistical analyses of the population of habitable planets as well as in-depth analyses of the climates of individual planets. Given the close relationship between habitability and surface liquid water, it is important to ask whether the fraction of a planets surface where water can be a liquid, $chi_text{hab}$, can be inferred from observations. We have produced a diverse bank of 1,874 3D climate models and computed the full-phase reflectance and emission spectrum for each model to investigate whether surface climate inference is feasible with high-quality direct imaging or secondary eclipse spectroscopy. These models represent the outcome of approximately 200,000 total simulated years of climate and over 50,000 CPU-hours, and the roughly-100 GB model bank and its associated spectra are being made publicly-available for community use. We find that there are correlations between spectra and $chi_text{hab}$ that will permit statistical approaches. However, spectral degeneracies in the climate observables produced by our model bank indicate that inference of individual climates is likely to be model-dependent, and inference will likely be impossible without exhaustive explorations of the climate parameter space. The diversity of potential climates on habitable planets therefore poses fundamental challenges to remote sensing efforts targeting exo-Earths.