In a microrheological set-up a single probe particle immersed in a complex fluid is exposed to a strong external force driving the system out of equilibrium. Here, we elaborate analytically the time-dependent response of a probe particle in a dilute suspension of Brownian particles to a large step-force, exact in first order of the density of the bath particles. The time-dependent drift velocity approaches its stationary state value exponentially fast for arbitrarily small driving in striking contrast to the power-law prediction of linear response encoded in the long-time tails of the velocity autocorrelation function. We show that the stationary-state behavior depends nonanalytically on the driving force and connect this behavior to the persistent correlations in the equilibrium state. We argue that this relation holds generically. Furthermore, we elaborate that the fluctuations in the direction of the force display transient superdiffusive behavior.