We compute the orbital angular momentum $L_z$ of an s-wave paired superfluid in the presence of an axisymmetric multiply quantized vortex. For vortices with winding number $|k| > 1$, we find that in the weak-pairing BCS regime $L_z$ is significantly reduced from its value $hbar N k/2$ in the Bose-Einstein condensation (BEC) regime, where $N$ is the total number of fermions. This deviation results from the presence of unpaired fermions in the BCS ground state, which arise as a consequence of spectral flow along the vortex sub-gap states. We support our results analytically and numerically by solving the Bogoliubov-de-Gennes equations within the weak-pairing BCS regime.