Cosmological perturbation theory for the late Universe dominated by dark matter is extended beyond the perfect fluid approximation by taking the dark matter velocity dispersion tensor as an additional field into account. A proper tensor decomposition of the latter leads to two additional scalar fields, as well as a vector and a tensor field. Most importantly, the trace of the velocity dispersion tensor can have a spatially homogeneous and isotropic expectation value. While it decays at early times, we show that a back-reaction effect quadratic in perturbations makes it grow strongly at late times. We compare sterile neutrinos as a candidate for comparatively warm dark matter to weakly interacting massive particles as a rather cold dark matter candidate and show that the late time growth of velocity dispersion is stronger for the latter. Another feature of a non-vanishing velocity dispersion expectation value is that it destroys the apparent self-consistency of the single-stream approximation and allows thereby to treat times and scales beyond shell-crossing.