Up to now there has been no reliable method to calculate the Casimir force when surface roughness becomes comparable with the separation between bodies. Statistical analysis of rough Au films demonstrates rare peaks with heights considerably larger than the root-mean-square (rms) roughness. These peaks define the minimal distance between rough surfaces and can be described with extreme value statistics. We show that the contributions of high peaks to the force can be calculated independently of each other while the contribution of normal roughness can be evaluated perturbatively beyond the proximity force approximation. The developed method allows a reliable force estimation for short separations. Our model explains the strong hitherto unexplained deviation from the normal Casimir scaling observed experimentally at short separations.