[abridged] Understanding how the infalling gas redistribute most of its initial angular momentum inherited from prestellar cores before reaching the stellar embryo is a key question. Disk formation has been naturally considered as a possible solution to this angular momentum problem. However, how the initial angular momentum of protostellar cores is distributed and evolves during the main accretion phase and the beginning of disk formation has largely remained unconstrained up to now. In the framework of the IRAM CALYPSO survey, we used high dynamic range C$^{18}$O (2-1) and N$_2$H$^+$ (1-0) observations to quantify the distribution of specific angular momentum along the equatorial axis in a sample of 12 Class 0 protostellar envelopes from scales ~50 to 10000 au. The radial distributions of specific angular momentum in the CALYPSO sample suggest two distinct regimes within protostellar envelopes: the specific angular momentum decreases as $j propto r^{1.6 pm 0.2}$ down to ~1600 au and then tends to become relatively constant around 6 $times$ 10$^{-4}$ km s$^{-1}$ pc down to ~50 au. The values of specific angular momentum measured in the inner Class 0 envelopes, namely that of the material directly involved in the star formation process ($<$1600 au), is on the same order of magnitude as what is inferred in small T-Tauri disks. Thus, disk formation appears to be a direct consequence of angular momentum conservation during the collapse. Our analysis reveals a dispersion of the directions of velocity gradients at envelope scales $>$1600 au, suggesting that they may not be related to rotational motions of the envelopes. We conclude that the specific angular momentum observed at these scales could find its origin in core-forming motions (infall, turbulence) or trace an imprint of the initial conditions for the formation of protostellar cores.