We show that almost all the linear differential operators factors obtained in the analysis of the n-particle contribution of the susceptibility of the Ising model for $, n le 6$, are operators associated with elliptic curves. Beyond the simplest factors which are homomorphic to symmetric powers of the second order operator associated with the complete elliptic integral E, the second and third order differential operators can actually be interpreted as modular forms of the elliptic curve of the Ising model. A last order-four globally nilpotent operator is not reducible to this elliptic curve, modular forms scheme. It is shown to actually correspond to a natural generalization of this elliptic curve, modular forms scheme, with the emergence of a Calabi-Yau equation, corresponding to a selected $_4F_3$ hypergeometric function which can also be seen as a Hadamard product of the complete elliptic integral K, with a remarkably simple algebraic pull-back, the corresponding Calabi-Yau fourth-order differential operator having a symplectic differential Galois group SP(4,C). The associated mirror maps and higher order Schwarzian ODEs has an exact (isogenies) representation of the generators of the renormalization group, extending the modular group SL(2,Z) to a GL(2, Z) symmetry group.