We investigate the condition for the formation of low-mass second-generation stars in the early universe. It has been proposed that gas cooling by dust thermal emission can trigger fragmentation of a low-metallicity star-forming gas cloud. In order to determine the critical condition in which dust cooling induces the formation of low-mass stars, we follow the thermal evolution of a collapsing cloud by a one-zone semi-analytic collapse model. Earlier studies assume the dust amount in the local universe, where all refractory elements are depleted onto grains, and/or assume the constant dust amount during gas collapse. In this paper, we employ the models of dust formation and destruction in early supernovae to derive the realistic dust compositions and size distributions for multiple species as the initial conditions of our collapse calculations. We also follow accretion of heavy elements in the gas phase onto dust grains, i.e., grain growth, during gas contraction. We find that grain growth well alters the fragmentation property of the clouds, and that this still does not approach to the value in the local universe. The critical conditions can be written by the gas metallicity Zcr and the initial depletion efficiency fdep,0 of gas-phase metal onto grains, or dust-to-metal mass ratio, as (Zcr/10^{-5.5} Zsun) = (fdep,0/0.18)^{-0.44} with small scatters in the range of Zcr = [0.06--3.2]x10^{-5} Zsun. We also show that the initial dust composition and size distribution are important to determine Zcr.