Grain growth during star formation affects the physical and chemical processes in the evolution of star-forming clouds. We investigate the origin of the millimeter (mm)-sized grains recently observed in Class I protostellar envelopes. We use the coagulation model developed in our previous paper and find that a hydrogen number density of as high as $10^{10}~{rm cm^{-3}}$, instead of the typical density $10^5~{rm cm^{-3}}$, is necessary for the formation of mm-sized grains. Thus, we test a hypothesis that such large grains are transported to the envelope from the inner, denser parts, finding that gas drag by outflow efficiently launches the large grains as long as the central object has not grown to $gtrsim 0.1$ M$_{odot}$. By investigating the shattering effect on the mm-sized grains, we ensure that the large grains are not significantly fragmented after being injected in the envelope. We conclude that the mm-sized grains observed in the protostellar envelopes are not formed in the envelopes but formed in the inner parts of the star-forming regions and transported to the envelopes before a significant mass growth of the central object, and that they survive in the envelopes.