We calculate the instanton partition function of the four-dimensional N=2* SU(N) gauge theory in the presence of a generic surface operator, using equivariant localization. By analyzing the constraints that arise from S-duality, we show that the effective twisted superpotential, which governs the infrared dynamics of the two-dimensional theory on the surface operator, satisfies a modular anomaly equation. Exploiting the localization results, we solve this equation in terms of elliptic and quasi-modular forms which resum all non-perturbative corrections. We also show that our results, derived for monodromy defects in the four-dimensional theory, match the effective twisted superpotential describing the infrared properties of certain two-dimensional sigma models coupled either to pure N=2 or to N=2* gauge theories.