The performance of millimeter wave (mmWave) communications critically depends on the accuracy of beamforming both at base station (BS) and user terminals (UEs) due to high isotropic path-loss and channel attenuation. In high mobility environments, accurate beam alignment becomes even more challenging as the angles of the BS and each UE must be tracked reliably and continuously. In this work, focusing on the beamforming at the BS, we propose an adaptive method based on Recurrent Neural Networks (RNN) that tracks and predicts the Angle of Departure (AoD) of a given UE. Moreover, we propose a modified frame structure to reduce beam alignment overhead and hence increase the communication rate. Our numerical experiments in a highly non-linear mobility scenario show that our proposed method is able to track the AoD accurately and achieve higher communication rate compared to more traditional methods such as the particle filter.