The chemical enrichment in the interstellar medium (ISM) of galaxies is regulated by several physical processes: stellar evolution, grain formation and destruction, galactic inflows and outflows. Understanding such processes is essential to follow the chemical enrichment of galaxies through the cosmic epochs, and to interpret the observations. Despite the importance of such topics, the efficiency of the different processes driving the evolution of baryons in galaxies, remain controversial. We revise the current description of metal and dust evolution in local low-metallicity dwarf galaxies and we develop a description for Lyman Break Galaxies. Our main goal is to reproduce i) the peak in the mass of dust over the mass of stars (sMdust) observed within few hundred Myrs; ii) the decrease of the sMdust at later time. The spectral energy distribution of the galaxies is fitted with the Code Investigating GALaxies Emission (CIGALE), through which the stellar and dust masses, and the star formation rate are estimated. For some of the dwarf galaxies, the metal and gas content are also available. We run different calculations of chemical evolution in galaxies, and we fit the observed properties through the model predictions. We show that i) a top-heavy initial mass function that favours massive stars and a dust condensation fraction for Type II Supernovae (SNe II) of 50% or more help to reproduce the peak of sMdust observed after 100 Myrs since the beginning of the cycle; ii) galactic outflows play a crucial role in reproducing the decline in sMdust with age, and they are more efficient than grain destruction from SNe II; iii) a star formation efficiency (mass of gas converted into stars) of few per cent is required to explain the metallicity of local dwarf galaxies; iv) dust growth in the ISM is not necessary to reproduce the sMdust and, if present, its effect is erased by galactic outflows.