Using neutron diffraction, we have studied the magnetic field effect on charge structures in the charge-ordered multiferroic material LuFe$_2$O$_4$. An external magnetic field is able to change the magnitude and correlation lengths of the charge valence order even before the magnetic order sets in. This affects the dielectric and ferroelectric properties of the material and induces a giant magneto-electric effect. Our results suggest that the magneto-electric coupling in LuFe$_2$O$_4$ is likely due to magnetic field effect on local spins, in clear contrast to the case in most other known multiferroic systems where the bulk magnetic order is important.