We present a hierarchical probabilistic model for improving geometric stellar distance estimates using color--magnitude information. This is achieved with a data driven model of the color--magnitude diagram, not relying on stellar models but instead on the relative abundances of stars in color--magnitude cells, which are inferred from very noisy magnitudes and parallaxes. While the resulting noise-deconvolved color--magnitude diagram can be useful for a range of applications, we focus on deriving improved stellar distance estimates relying on both parallax and photometric information. We demonstrate the efficiency of this approach on the 1.4 million stars of the Gaia TGAS sample that also have APASS magnitudes. Our hierarchical model has 4~million parameters in total, most of which are marginalized out numerically or analytically. We find that distance estimates are significantly improved for the noisiest parallaxes and densest regions of the color--magnitude diagram. In particular, the average distance signal-to-noise ratio and uncertainty improve by 19~percent and 36~percent, respectively, with 8~percent of the objects improving in SNR by a factor greater than 2. This computationally efficient approach fully accounts for both parallax and photometric noise, and is a first step towards a full hierarchical probabilistic model of the Gaia data.