It has been recently claimed that cosmologies with time dependent speed of light might solve some of the problems of the standard cosmological scenario, as well as inflationary scenarios. In this letter we show that most of these models, when analyzed in a consistent way, lead to large violations of charge conservation. Thus, they are severly constrained by experiment, including those where $c$ is a power of the scale factor and those whose source term is the trace of the energy-momentum tensor. In addition, early Universe scenarios with a sudden change of $c$ related to baryogenesis are discarded.