We study the empirical realisation of the memory effect in Yang-Mills theory, especially in view of the classical vs. quantum nature of the theory. Gauge invariant analysis of memory in classical U(1) electrodynamics and its observation by total change of transverse momentum of a charge is reviewed. Gauge fixing leads to a determination of a gauge transformation at infinity. An example of Yang-Mills memory then is obtained by reinterpreting known results on interactions of a quark and a large high energy nucleus in the theory of Color Glass Condensate. The memory signal is again a kick in transverse momentum, but it is only obtained in quantum theory after fixing the gauge, after summing over an ensemble of classical processes.