We have performed simultaneous measurements of magnetic chirality by using polarized neutrons and electric polarization along the b-axis of single crystals of YMn$^{4+}$(Mn$_{1-x}$Ga$_{x}$)$^{3+}$O$_{5}$ with $x=0.047$ and 0.12, in which nonmagnetic Ga-ions dilute Mn$^{3+}$ spins. The $x=0.047$ sample exhibits high-temperature incommensurate (HT-ICM), commensurate (CM), and low-temperature incommensurate (LT-ICM) magnetic phases in order of decreasing temperature, whereas the $x=0.12$ sample exhibits only HT-ICM and LT-ICM phases. Here, the CM and LT-ICM phases are ferroelectric and weak-ferroelectric, respectively. Measurements conducted under zero field heating after various field-cooling conditions evidence that the microscopic mechanisms of the spin-driven ferroelectricity in the CM and LT-ICM phases are different: the magnetic chirality of Mn$^{4+}$ cycloidal spins plays a dominant role in the LT-ICM phase, whereas the magnetic exchange striction by the Mn$^{4+}$-Mn$^{3+}$ chain plays a dominant role in the CM phase. The polarization of YMn$_{2}$O$_{5}$ flips upon CM to LT-ICM phase transition because the ferroelectricity driven by the magnetic chirality and the exchange striction provides opposite directions of polarization.