The motion of an optically trapped sphere constrained by the vicinity of a wall is investigated at times where hydrodynamic memory is significant. First, we quantify, in bulk, the influence of confinement arising from the trapping potential on the spheres velocity autocorrelation function $C(t)$. Next, we study the splitting of $C(t)$ into $C_parallel(t)$ and $C_perp(t)$, when the sphere is approached towards a surface. Thereby, we monitor the crossover from a slow $t^{-3/2}$ long-time tail, away from the wall, to a faster $t^{-5/2}$ decay, due to the subtle interplay between hydrodynamic backflow and wall effects. Finally, we discuss the resulting asymmetric time-dependent diffusion coefficients.