We report on the synthesis and physical properties of cm-sized CoGeO$_3$ single crystals grown in a high pressure mirror furnace at pressures of 80~bar. Direction dependent magnetic susceptibility measurements on our single crystals reveal highly anisotropic magnetic properties that we attribute to the impact of strong single ion anisotropy appearing in this system with T$_N$~$sim$~33.5~K. Furthermore, we observe effective magnetic moments that are exceeding the spin only values of the Co ions which reveals the presence of sizable orbital moments in CoGeO$_3$.