The Al/Si and Mg/Si ratios in non-carbonaceous chondrites are lower than the solar (i.e., CI-chondritic) values, in sharp contrast to the non-CI carbonaceous meteorites and the Earth, which are enriched in refractory elements and have Mg/Si ratios that are solar or larger. We show that the formation of a first generation of planetesimals during the condensation of refractory elements implies the subsequent formation of residual condensates with strongly sub-solar Al/Si and Mg/Si ratios. The mixing of residual condensates with different amounts of material with solar refractory/Si element ratios explains the Al/Si and Mg/Si values of non-carbonaceous chondrites. To match quantitatively the observed ratios, we find that the first-planetesimals should have accreted when the disk temperature was ~1,330-1,400 K depending on pressure and assuming a solar C/O ratio of the disk. We discuss how this model relates to our current understanding of disk evolution, grain dynamics, and planetesimal formation. We also extend the discussion to moderately volatile elements (e.g., Na), explaining how it may be possible that the depletion of these elements in non-carbonaceous chondrites is correlated with the depletion of refractory elements (e.g., Al). Extending the analysis to Cr, we find evidence for a higher than solar C/O ratio in the protosolar disks gas when/where condensation from a fractionated gas occurred. Finally, we discuss the possibility that the supra-solar Al/Si and Mg/Si ratios of the Earth are due to the accretion of ~40% of the mass of our planet from the first-generation of refractory-rich planetesimals.