We study the relationship between the structure and star-formation rate (SFR) of X-ray selected low and moderate luminosity active galactic nuclei (AGNs) in the two Chandra Deep Fields, using Hubble Space Telescope imaging from the Cosmic Assembly Near Infrared Extragalactic Legacy Survey (CANDELS) and deep far-infrared maps from the PEP+GOODS-Herschel survey. We derive detailed distributions of structural parameters and FIR luminosities from carefully constructed control samples of galaxies, which we then compare to those of the AGNs. At z~1, AGNs show slightly diskier light profiles than massive inactive (non-AGN) galaxies, as well as modestly higher levels of gross galaxy disturbance (as measured by visual signatures of interactions and clumpy structure). In contrast, at z~2, AGNs show similar levels of galaxy disturbance as inactive galaxies, but display a red central light enhancement, which may arise due to a more pronounced bulge in AGN hosts or due to extinguished nuclear light. We undertake a number of tests of these alternatives, but our results do not strongly favour one interpretation over the other. The mean SFR and its distribution among AGNs and inactive galaxies are similar at z>1.5. At z<1, however, clear and significant enhancements are seen in the SFRs of AGNs with bulge-dominated light profiles. These trends suggest an evolution in the relation between nuclear activity and host properties with redshift, towards a minor role for mergers and interactions at z>1.5.