We present a renormalizable fermion mass model based on the symmetry $Q_4$ that accommodates all fermion masses and mixing angles in both the quark and lepton sectors. It requires the presence of only four SU(2) doublet scalar fields transforming non trivially under the flavor symmetry and the assumption of an alignment between first and second generation Yukawa couplings. No right-handed neutrinos are present in the model and neutrino masses are generated radiatively through the introduction of two additional SU(2) singlet fields charged under both hypercharge and lepton number.