Impurity scattering is found to lead to quasi-one dimensional nanoscale modulation of the local density of states in the iron pnictides and chalcogenides. This `quasiparticle interference feature is remarkably similar across a wide variety of pnictide and chalcogenide phases, suggesting a common origin. We show that a unified understanding of the experiments can be obtained by simply invoking a four-fold symmetry breaking $d_{xz}-d_{yz}$ orbital splitting, of a magnitude already suggested by the experiments. This can explain the one-dimensional characteristics in the local density of states observed in the orthorhombic nematic, tetragonal paramagnetic, as well as the spin-density wave and superconducting states in these materials.