In this study, we carry out density functional theory calculations to elucidate the polarization switching mechanism in charge-order-induced ferroelectrics. Based on the investigations about (SrVO$_3$)$_1$(LaVO$_3$)$_1$ superlattice, we demonstrate that the charge ordering state couples strongly to lattice modes, and charge-transfer induced polarization switching has to be associated with changes of lattice distortions. Based on the type of lattice mode strongly coupled to charge ordering states, we classify the charge ordering materials in two type, namely polyhedral breathing and off-centering displacement types. We further demonstrate that only in off-centering displacement type charger ordering material, the polarization is switchable under an external field. The implications of this theory to experimental observations are also discussed and we successfully explain the different electrical behaviors in LuFe$_2$O$_4$ and Fe$_3$O$_4$. This study aims to provide guidance for searching and designing charge ordering ferroelectrics with switchable polarization.