Control of the motion of cavity solitons is one the central problems in nonlinear optical pattern formation. We report on the impact of the phase of the time-delayed optical feedback and carrier lifetime on the self-mobility of localized structures of light in broad area semiconductor cavities. We show both analytically and numerically that the feedback phase strongly affects the drift instability threshold as well as the velocity of cavity soliton motion above this threshold. In addition we demonstrate that non-instantaneous carrier response in the semiconductor medium is responsible for the increase in critical feedback rate corresponding to the drift instability.