Experiments searching for the electric dipole moment (EDM) of the electron $d_e$ utilise atomic/molecular states with one or more uncompensated electron spins, and these paramagnetic systems have recently achieved remarkable sensitivity to $d_e$. If the source of $CP$ violation resides entirely in the hadronic sector, the two-photon exchange processes between electrons and the nucleus induce $CP$-odd semileptonic interactions, parametrised by the Wilson coefficient $C_{SP}$, and provide the dominant source of EDMs in paramagnetic systems instead of $d_e$. We evaluate the $C_{SP}$ coefficients induced by the leading hadronic sources of $CP$ violation, namely nucleon EDMs and $CP$-odd pion-nucleon couplings, by calculating the nucleon-number-enhanced $CP$-odd nuclear scalar polarisability, employing chiral perturbation theory at the nucleon level and the Fermi-gas model for the nucleus. This allows us to translate the ACME EDM limits from paramagnetic ThO into novel independent constraints on the QCD theta term $|bar theta| < 3 times 10^{-8}$, proton EDM $|d_p| < 2 times 10^{-23},e,{rm cm}$, isoscalar $CP$-odd pion-nucleon coupling $|bar g^{(1)}_{pi NN}| < 4 times 10^{-10}$, and colour EDMs of quarks $|tilde d_u - tilde d_d| < 2 times 10^{-24},{rm cm}$. We note that further experimental progress with EDM experiments in paramagnetic systems may allow them to rival the sensitivity of EDM experiments with neutrons and diamagnetic atoms to these quantities.