For an arbitrary possibly non-Hermitian matrix Hamiltonian H, that might involve exceptional points, we construct an appropriate parameter space M and the lines bundle L^n over M such that the adiabatic geometric phases associated with the eigenstates of the initial Hamiltonian coincide with the holonomies of L^n. We examine the case of 2 x 2 matrix Hamiltonians in detail and show that, contrary to claims made in some recent publications, geometric phases arising from encircling exceptional points are generally geometrical and not topological in nature.