We discuss the modular anomaly equation satisfied by the the prepotential of 4-dimensional N=2* theories and show that its validity is related to S-duality. The recursion relations that follow from the modular anomaly equation allow one to write the prepotential in terms of (quasi)-modular forms, thus resumming the instanton contributions. These results can be checked against the microscopic multi-instanton calculus in the case of classical algebras, but are valid also for the exceptional E6, E7, E8, F4 and G2 algebras, where direct computations are not available.