We study an extension of the minimal gauged $L_{mu}-L_{tau}$ model in order to explain the anomalous magnetic moments of muon and electron simultaneously. Presence of an additional scalar doublet $eta$ and an in-built $Z_2$ symmetry under which the right handed singlet fermions and $eta$ are odd, leads to light neutrino mass in scotogenic fashion along with a stable dark matter candidate. In spite of the possibility of having positive and negative contributions to $(g-2)$ from vector boson and charged scalar loops respectively, the minimal scotogenic $L_{mu}-L_{tau}$ model can not explain muon and electron $(g-2)$ simultaneously while being consistent with other experimental bounds. We then extend the model with a vector like lepton doublet which not only leads to a chirally enhanced negative contribution to electron $(g-2)$ but also leads to the popular singlet-doublet fermion dark matter scenario. With this extension, the model can explain both electron and muon $(g-2)$ while being consistent with neutrino mass, dark matter and other direct search bounds. The model remains predictive at high energy experiments like collider as well as low energy experiments looking for charged lepton flavour violation, dark photon searches, in addition to future $(g-2)$ measurements.