We group materials into five symmetry classes and determine in which of these classes phonons carry angular momentum in the Brillouin zone, away from a high-symmetry point, line, or plane. In some materials phonons acquire angular momentum via the forces induced by relative displacements of atoms out of their equilibrium positions. However, for other materials, such as ferromagnetic iron, phonon angular momentum arises from the forces induced by relative velocities of atoms. These effects are driven by the spin-orbit interaction.