We present a combined experimental and theoretical study on the local magnetism of the Co ions in the spin-chain compound CoV2O6, which crystallizes in two different allotropic phases, alpha- and gamma-CoV2O6. Using x-ray magnetic circular dichroism, we have found a very large and a moderate orbital contribution to the magnetism in alpha- and gamma-CoV2O6, respectively. Full-multiplet calculations indicate that the differences in the magnetic behavior of alpha- and gamma-CoV2O6 phases originate from different local distortions of the CoO6 octahedra. In particular, the strong compression of the CoO6 octahedra in alpha-CoV2O6 lead to a strong mixture of t2g and eg orbitals which, via the local atomic Coulomb and exchange interactions, results in an exceptionally large orbital moment.