The relaxation dynamics of hot carriers in silicon (100) is studied via a novel holistic approach based on phase-resolved transient absorption spectroscopy with few-cycle optical pulses. After excitation by a sub-5 fs light pulse, strong electron-phonon coupling leads to an ultrafast momentum relaxation with time constant of 10 fs. The thermalization of the hot carriers occurs on a time constant of 150 fs, visible in the temporal evolution of the collision time as extracted from the Drude model. We find an increase of the collision time from 3 fs for the shortest timescales with a saturation at approximately 18 fs. Moreover, the optical effective mass of the hot carrier ensemble evolves on ultrafast timescales as well, with a bi-exponential decrease from 0.7 $m_e$ to about 0.125 $m_e$ and time constants of 4 fs and 58 fs. The presented information on the electron mass dynamics as well as the momentum-, energy-, and collision-scattering times with unprecedented time resolution is important for all hot carrier optoelectronic devices.