We argue that a complete description of quantum annealing (QA) implemented with continuous variables must take into account the non-adiabatic Aharonov-Anandan geometric phase that arises when the system Hamiltonian changes during the anneal. We show that this geometric effect leads to the appearance of non-stoquastic terms in the effective quantum Ising Hamiltonians that are typically used to describe QA with flux-qubits. We explicitly demonstrate the effect of these geometric interactions when QA is performed with a system of one and two coupled flux-qubits. The realization of non-stoquastic Hamiltonians has important implications from a computational complexity perspective, since it is believed that in many cases QA with stoquastic Hamiltonians can be efficiently simulated via classical algorithms such as Quantum Monte Carlo. It is well-known that the direct implementation of non-stoquastic interactions with flux-qubits is particularly challenging. Our results suggest an alternative path for the implementation of non-stoquastic interactions via geometric phases that can be exploited for computational purposes.