The phase of quantum magneto-oscillations is often associated with the Berry phase and is widely used to argue in favor of topological nontriviality of the system (Berry phase $2pi n+pi$). Nevertheless, the experimentally determined value may deviate from $2pi n+pi$ arbitrarily, therefore more care should be made analyzing the phase of magneto-oscillations to distinguish trivial systems from nontrivial. In this paper we suggest two simple mechanisms dramatically affecting the experimentally observed value of the phase in three-dimensional topological insulators: (i) magnetic field dependence of the chemical potential, and (ii) possible nonuniformity of the system. These mechanisms are not limited to topological insulators and can be extended to other topologically trivial and non-trivial systems.