Kerr/CFT correspondence has been recently applied to various types of 5D extremal rotating black holes. A common feature of all such examples is the existence of two chiral CFT duals corresponding to the U(1) symmetries of the near horizon geometry. In this paper, by studying the moduli space of the near horizon metric of five dimensional extremal black holes which are asymptotically flat or AdS, we realize an SL(2,Z) modular group which is a symmetry of the near horizon geometry. We show that there is a lattice of chiral CFT duals corresponding to the moduli points identified under the action of the modular group. The microscopic entropy corresponding to all such CFTs are equivalent and are in agreement with the Bekenstein-Hawking entropy.