The inversion of gravitational lens systems is hindered by the fact that multiple mass distributions are often equally compatible with the observed properties of the images. Besides using clear examples to illustrate the effect of the so-called monopole and mass sheet degeneracies, this article introduces the most general form of said mass sheet degeneracy. While the well known version of this degeneracy rescales a single source plane, this generalization allows any number of sources to be rescaled. Furthermore, it shows how it is possible to rescale each of those sources with a different scale factor. Apart from illustrating that the mass sheet degeneracy is not broken by the presence of multiple sources at different redshifts, it will become apparent that the newly constructed mass distribution necessarily alters the existing mass density precisely at the locations of the images in the lens system, and that this change in mass density is linked to the factors with which the sources were rescaled. Combined with the fact that the monopole degeneracy introduces a large amount of uncertainty about the density in between the images, this means that both degeneracies are in fact closely related to substructure in the mass distribution. An example simulated lensing situation based on an elliptical version of a Navarro-Frenk-White profile explicitly shows that such degeneracies are not easily broken by observational constraints, even when multiple sources are present. Instead, the fact that each lens inversion method makes certain assumptions, implicit or explicit, about the smoothness of the mass distribution means that in practice the degeneracies are broken in an artificial manner rather than by observed properties of the lens system.