We here report coherent reflection of thermal He atom beams from various microscopically rough surfaces at grazing incidence. For a sufficiently small normal component $k_z$ of the incident wave-vector of the atom the reflection probability is found to be a function of $k_z$ only. This behavior is explained by quantum-reflection at the attractive branch of the Casimir-van der Waals interaction potential. For larger values of $k_z$ the overall reflection probability decreases rapidly and is found to also depend on the parallel component $k_x$ of the wave-vector. The material specific $k_x$ dependence for this classical reflection at the repulsive branch of the potential is explained qualitatively in terms of the averaging-out of the surface roughness under grazing incidence conditions.