We present evidence that band gap narrowing at the heterointerface may be a major cause of the large open circuit voltage deficit of Cu$_2$ZnSnS$_4$/CdS solar cells. Band gap narrowing is caused by surface states that extend the Cu$_2$ZnSnS$_4$ valence band into the forbidden gap. Those surface states are consistently found in Cu$_2$ZnSnS$_4$, but not in Cu$_2$ZnSnSe$_4$, by first-principles calculations. They do not simply arise from defects at surfaces but are an intrinsic feature of Cu$_2$ZnSnS$_4$ surfaces. By including those states in a device model, the outcome of previously published temperature-dependent open circuit voltage measurements on Cu$_2$ZnSnS$_4$ solar cells can be reproduced quantitatively without necessarily assuming a cliff-like conduction band offset with the CdS buffer layer. Our first-principles calculations indicate that Zn-based alternative buffer layers are advantageous due to the ability of Zn to passivate those surface states. Focusing future research on Zn-based buffers is expected to significantly improve the open circuit voltage and efficiency of pure-sulfide Cu$_2$ZnSnS$_4$ solar cells.