The recent discovery of high redshift dusty galaxies implies a rapid dust enrichment of their interstellar medium (ISM). To interpret these observations, we run a cosmological simulation in a 30$h^{-1}$ cMpc/size volume down to $z approx 4$. We use the hydrodynamical code dustyGadget, which accounts for the production of dust by stellar populations and its evolution in the ISM. We find that the cosmic dust density parameter ($Omega_{rm d}$) is mainly driven by stellar dust at $z gtrsim 10$, so that mass- and metallicity-dependent yields are required to assess the dust content in the first galaxies. At $z lesssim 9$ the growth of grains in the ISM of evolved systems (Log$(M_{star}/M_{odot})>8.5$) significantly increases their dust mass, in agreement with observations in the redshift range $4 lesssim z < 8$. Our simulation shows that the variety of high redshift galaxies observed with ALMA can naturally be accounted for by modeling the grain-growth timescale as a function of the physical conditions in the gas cold phase. In addition, the trends of dust-to-metal (DTM) and dust-to-gas (${cal D}$) ratios are compatible with the available data. A qualitative investigation of the inhomogeneous dust distribution in a representative massive halo at $z approx 4$ shows that dust is found from the central galaxy up to the closest satellites along polluted filaments with $rm Log({cal D}) leq -2.4$, but sharply declines at distances $d gtrsim 30$ kpc along many lines of sight, where $rm Log({cal D}) lesssim -4.0$.