We study the transfer of angular momentum in high energy nuclear collisions from the colliding nuclei to the region around midrapidity, using the classical approximation of the Color Glass Condensate (CGC) picture. We find that the angular momentum shortly after the collision (up to times ~ 1/Q_s, where Q_s is the saturation scale) is carried by the beta-type flow of the initial classical gluon field, introduced by some of us earlier. beta^i ~ mu_1 nabla^i mu_2 - mu_2 nabla^i mu_1 (i=1,2) describes the rapidity-odd transverse energy flow and emerges from Gauss Law for gluon fields. Here mu_1 and mu_2 are the averaged color charge fluctuation densities in the two nuclei, respectively. Interestingly, strong coupling calculations using AdS/CFT techniques also find an energy flow term featuring this particular combination of nuclear densities. In classical CGC the order of magnitude of the initial angular momentum per rapidity in the reaction plane, at a time 1/Q_s, is |dL_2/d eta| ~ R_A/Q_s^3 epsilon_0/2 at midrapidity, where R_A is the nuclear radius, and epsilon_0 is the average initial energy density. This result emerges as a cancellation between a vortex of energy flow in the reaction plane aligned with the total angular momentum, and energy shear flow opposed to it. We discuss in detail the process of matching classical Yang-Mills results to fluid dynamics. We will argue that dissipative corrections should not be discarded to ensure that macroscopic conservation laws, e.g. for angular momentum, hold. Viscous fluid dynamics tends to dissipate the shear flow contribution that carries angular momentum in boost-invariant fluid systems. This leads to small residual angular momentum around midrapidity at late times for collisions at high energies.