Semi-permeable membranes are important elements in water purification and energy generation applications, for which the atomic thickness and strength of graphene can enhance efficiency and permeation rate while maintaining good selectivity. Here, we show that an osmotic pressure difference forms across a suspended graphene membrane as a response to a sucrose concentration difference, providing evidence for its semi-permeability. This osmotic pressure difference is detected via the deflection of the graphene membrane that is measured by atomic force microscopy. Using this technique, the time dependence of this deflection allows us to measure the water permeation rate of a single 3.4 $mu$m diameter graphene membrane. Its value is close to the expected value of a single nanopore in graphene. The method thus allows one to experimentally study the semi-permeability of graphene membranes at the microscale when the leakage rate is miniscule. It can therefore find use in the development of graphene membranes for filtration, and can enable sensors that measure the concentration and composition of solutions.