We investigated fundamental processes of collisional sticking and fragmentation of dust aggregates by carrying out N-body simulations of submicron-sized icy dust monomers. We examined the condition for collisional growth of two colliding dust aggregates in a wide range of the mass ratio, 1-64. We found that the mass transfer from a larger dust aggregate to a smaller one is a dominant process in collisions with a mass ratio of 2-30 and impact velocity of approx 30-170 m s^-1. As a result, the critical velocity, v_fra, for fragmentation of the largest body is considerably reduced for such unequal-mass collisions; v_fra of collisions with a mass ratio of 3 is about half of that obtained from equal-mass collisions. The impact velocity is generally higher for collisions between dust aggregates with higher mass ratios because of the difference between the radial drift velocities in the typical condition of protoplanetary disks. Therefore, the reduced v_fra for unequal-mass collisions would delay growth of dust grains in the inner region of protoplanetary disks.