Although pinning of domain walls in ferromagnets is ubiquitous, the absence of an appropriate characterization tool has limited the ability to correlate the physical and magnetic microstructures of ferromagnetic films with specific pinning mechanisms. Here, we show that the pinning of a magnetic vortex, the simplest possible domain structure in soft ferromagnets, is strongly correlated with surface roughness, and we make a quantitative comparison of the pinning energy and spatial range in films of various thickness. The results demonstrate that thickness fluctuations on the lateral length scale of the vortex core diameter, i.e. an effective roughness at a specific length scale, provides the dominant pinning mechanism. We argue that this mechanism will be important in virtually any soft ferromagnetic film.