The extragalactic distance scale builds on the Cepheid period-luminosity (PL) relation. In this paper, we want to carry out a strictly differential comparison of the absolute PL relations obeyed by classical Cepheids in the Milky Way (MW), LMC and SMC galaxies. Taking advantage of the substantial metallicity difference among the Cepheid populations in these three galaxies, we want to establish a possible systematic trend of the PL relation absolute zero point as a function of metallicity, and determine the size of such an effect in optical and near-infrared photometric bands. We are using the IRSB Baade-Wesselink type method as calibrated by Storm et al. to determine individual distances to the Cepheids in our samples in MW, LMC and SMC. For our analysis, we use a greatly enhanced sample of Cepheids in the SMC (31 stars) as compared to the small sample (5 stars) available in our previous work. We use the distances to determine absolute Cepheid PL relations in optical and near-infrared bands in each of the three galaxies.} {Our distance analysis of 31 SMC Cepheids with periods from 4-69 days yields tight PL relations in all studied bands, with slopes consistent with the corresponding LMC and MW relations. Adopting the very accurately determined LMC slopes for the optical and near-infrared bands, we determine the zero point offsets between the corresponding absolute PL relations in the 3 galaxies. We find that in all bands the metal-poor SMC Cepheids are intrinsically fainter than their more metal-rich counterparts in the LMC and MW. In the $K$ band the metallicity effect is $-0.23pm0.06$~mag/dex while in the $V,(V-I)$ Wesenheit index it is slightly stronger, $-0.34pm0.06$~mag/dex. We find some evidence that the PL relation zero point-metallicity relation might be nonlinear, becoming steeper for lower metallicities.