The voltage-controlled Berry phases in two vertically coupled InGaAs/GaAs quantum dots are investigated theoretically. It is found that Berry phases can be changed dramatically from 0 to 2$pi$ (or 2$pi$ to 0) only simply by turning the external voltage. Under realistic conditions, as the tunneling is varied from $0.8eV$ to $0.9eV$ via a bias voltage, the Berry phases are altered obviously, which can be detected in an interference experiment. The scheme is expected to be useful in constructing quantum computation based on geometric phases in an asymmetrical double quantum dot controlled by voltage.