Let P be a lattice polytope with $h^*$-vector $(1, h^*_1, h^*_2)$. In this note we show that if $h_2^* leq h_1^*$, then $P$ is IDP. More generally, we show the corresponding statements for semi-standard graded Cohen-Macaulay domains over algebraically closed fields.