Using deep 100-160 micron observations in GOODS-S from the GOODS-H survey, combined with HST/WFC3 NIR imaging from CANDELS, we present the first morphological analysis of a complete, FIR selected sample of 52 ULIRGs at z~2. We also make use of a comparison sample of galaxies without Herschel detections but with the same z and magnitude distribution. Our visual classifications of these two samples indicate that the fraction of objects with disk and spheroid morphologies is roughly the same but that there are significantly more mergers, interactions, and irregular galaxies among the ULIRGs. The combination of disk and irregular/interacting morphologies suggests that early stage interactions and minor mergers could play an important role in ULIRGs at z~2. We compare these fractions with those of a z~1 sample across a wide luminosity range and find that the fraction of disks decreases systematically with L_IR while the fraction of mergers and interactions increases, as has been observed locally. At comparable luminosities, the fraction of ULIRGs with various morphological classifications is similar at z~2 and z~1. We investigate the position of the ULIRGs, along with 70 LIRGs, on the specific star formation rate versus redshift plane, and find 52 systems to be starbursts (lie more than a factor of 3 above the main sequence relation). The morphologies of starbursts are dominated by interacting and merging systems (50%). If irregular disks are included as potential minor mergers, then we find that up to 73% of starbursts are involved in a merger or interaction at some level. Although the final coalescence of a major merger may not be required for the high luminosities of ULIRGs at z~2 as is the case locally, the large fraction of interactions at all stages and potential minor mergers suggest that the high star formation rates of ULIRGs are still largely externally triggered at z~2.