We present a new method of measuring optical near-fields within ~1 nm of a metal surface, based on rescattering of photoemitted electrons. With this method, we precisely measure the field enhancement factor for tungsten and gold nanotips as a function of tip radius. The agreement with Maxwell simulations is very good. Further simulations yield a field enhancement map for all materials, which shows that optical near-fields at nanotips are governed by a geometric effect under most conditions, while plasmon resonances play only a minor role. Last, we consider the implications of our results on quantum mechanical effects near the surface of nanostructures and discuss features of quantum plasmonics.