A nonzero electric dipole moment (EDM) of the neutron, proton, deuteron or helion, in fact, of any finite system necessarily involves the breaking of a symmetry, either by the presence of external fields (i.e. electric fields leading to the case of induced EDMs) or explicitly by the breaking of the discrete parity and time-reflection symmetries in the case of permanent EDMs. We discuss two theorems describing these phenomena and report about the cosmological motivation for an existence of CP breaking beyond what is generated by the Kobayashi-Maskawa mechanism in the Standard Model and what this might imply for the permanent electric dipole moments of the nucleon and light nuclei by estimating a window of opportunity for physics beyond what is currently known. Recent - and in the case of the deuteron even unpublished - results for the relevant matrix elements of nuclear EDM operators are presented and the relevance for disentangling underlying New Physics sources are discussed.