We examine correlated electron and doubly charged ion momentum spectra from strong field double ionization of Neon employing intense elliptically polarized laser pulses. An ellipticity-dependent asymmetry of correlated electron and ion momentum distributions has been observed. Using a 3D semiclassical model, we demonstrate that our observations reflect the sub-cycle dynamics of the recollision process. Our work reveals a general physical picture for recollision-impact double ionization with elliptical polarization, and demonstrates the possibility of ultrafast control of the recollision dynamics.