We report density functional calculations of the electronic structure and Fermi surface of the BaFe$_2$As$_2$ and LiFeAs phases including doping via the virtual crystal approximation. The results show that contrary to a rigid band picture, the density of states at the Fermi energy is only weakly doping dependent and that the main effect of doping is a change in the relative sizes of the electron and hole Fermi surfaces as required by Luttingers theory. This is a consequence of a change in As height with doping, in particular a shift of As towards Fe as holes are introduced in the Fe plane, as might be expected from simple ionic considerations. The main effect of doping is therefore a reduction in the degree of nesting of the Fermi surface. This provides a framework for understanding the approximate electron-hole symmetry in the phase diagrams of the Fe-As based superconductors.