The comparative study of the substitution of zinc and lithium for iron in the 114 ferrites, YBaFe4O7 and CaBaFe4O7, shows that these diamagnetic cations play a major role in tuning the competition between ferrimagnetism and magnetic frustration in these oxides. The substitution of Li or Zn for Fe in the cubic phase YBaFe4O7 leads to a structural transition to a hexagonal phase YBaFe4-xMxO7, for M = Li (0.30 < x < 0.75) and for M = Zn (0.40 < x < 1.50). It is seen that for low doping values i.e. x = 0.30 (for Li) and x = 0.40 (for Zn), these diamagnetic cations induce a strong ferrimagnetic component in the samples, in contrast to the spin glass behaviour of the cubic phase. In all the hexagonal phases, YBaFe4-xMxO7 and CaBaFe4-xMxO7 with M = Li and Zn, it is seen that in the low doping regime (x ~ 0.3 to 0.5), the competition between ferrimagnetism and 2 D magnetic frustration is dominated by the average valency of iron. In contrast, in the high doping regime (x ~ 1.5), the emergence of a spin glass is controlled by the high degree of cationic disorder, irrespective of the iron valency.