We constrain the average halo ellipticity of ~2 600 galaxy groups from the Galaxy And Mass Assembly (GAMA) survey, using the weak gravitational lensing signal measured from the overlapping Kilo Degree Survey (KiDS). To do so, we quantify the azimuthal dependence of the stacked lensing signal around seven different proxies for the orientation of the dark matter distribution, as it is a priori unknown which one traces the orientation best. On small scales, the major axis of the brightest group/cluster member (BCG) provides the best proxy, leading to a clear detection of an anisotropic signal. In order to relate that to a halo ellipticity, we have to adopt a model density profile. We derive new expressions for the quadrupole moments of the shear field given an elliptical model surface mass density profile. Modeling the signal with an elliptical Navarro-Frenk-White (NFW) profile on scales < 250 kpc, which roughly corresponds to half the virial radius, and assuming that the BCG is perfectly aligned with the dark matter, we find an average halo ellipticity of e_h=0.38 +/- 0.12. This agrees well with results from cold-dark-matter-only simulations, which typically report values of e_h ~ 0.3. On larger scales, the lensing signal around the BCGs does not trace the dark matter distribution well, and the distribution of group satellites provides a better proxy for the halos orientation instead, leading to a 3--4 sigma detection of a non-zero halo ellipticity at scales between 250 kpc and 750 kpc. Our results suggest that the distribution of stars enclosed within a certain radius forms a good proxy for the orientation of the dark matter within that radius, which has also been observed in hydrodynamical simulations.