The baroclinic annular mode (BAM) is a leading-order mode of the eddy-kinetic energy in the Southern Hemisphere exhibiting. oscillatory behavior at intra-seasonal time scales. The oscillation mechanism has been linked to transient eddy-mean flow interactions that remain poorly understood. Here we demonstrate that the finite memory effect in eddy-heat flux dependence on the large-scale flow can explain the origin of the BAMs oscillatory behavior. We represent the eddy memory effect by a delayed integral kernel that leads to a generalized Langevin equation for the planetary-scale heat equation. Using a mathematical framework for the interactions between planetary and synoptic-scale motions, we derive a reduced dynamical model of the BAM - a stochastically-forced oscillator with a period proportional to the geometric mean between the eddy-memory time scale and the diffusive eddy equilibration timescale. Our model provides a formal justification for the previously proposed phenomenological model of the BAM and could be used to explicitly diagnose the memory kernel and improve our understanding of transient eddy-mean flow interactions in the atmosphere.