Ne atoms with energies up to 3 keV are diffracted under grazing angles of incidence from a LiF(001) surface. For a small momentum component of the incident beam perpendicular to the surface, we observe an increase of the elastic rainbow angle together with a broadening of the inelastic scattering profile. We interpret these two effects as the refraction of the atomic wave in the attractive part of the surface potential. We use a fast, rigorous dynamical diffraction calculation to find a projectile-surface potential model that enables a quantitative reproduction of the experimental data for up to ten diffraction orders. This allows us to extract an attractive potential well depth of 10.4 meV. Our results set a benchmark for more refined surface potential models which include the weak Van der Waals region, a long-standing challenge in the study of atom-surface interactions.