We study the Affleck-Dine (AD) baryogenesis in the inflating curvaton scenario, when the curvaton is a moduli field with O($10-10^2$TeV) mass. A moduli field with such mass is known to be free from the Polonyi problem, and furthermore its decay products can explain the present cold dark matter abundance. In our scenario, it further explains the primordial curvature perturbation and the present baryon density all together. The current observational bound on the baryon isocurvature perturbation, which severely constrains the AD baryogenesis with the original oscillating moduli curvaton scenario, is shown to put practically negligible constraint if we replace the oscillating curvaton with the inflating curvaton.