We present a host morphological study of 1,265 far-infrared galaxies (FIRGs) and sub-millimeter galaxies (SMGs) in the Cosmic Evolution Survey field using the F160W and F814W images obtained by the Hubble Space Telescope. The FIRGs and the SMGs are selected from the Herschel Multi-tiered Extragalactic Survey and the SCUBA-2 Cosmology Legacy Survey, respectively. Their precise locations are based on the interferometry data from the Atacama Large Millimeter/submillimeter Array and the Very Large Array. The vast majority of these objects are at $0.1lesssim zlesssim 3$. While the SMGs do not constitute a subset of the FIRGs in our selection due to the signal-to-noise ratio thresholds, SMGs can be regarded as the population at the high-redshift tail of FIRGs. Most of our FIRGs/SMGs have total infrared luminosity ($L_{rm IR}$) in the regimes of luminous and ultra-luminous infrared galaxies (LIRGs, $L_{rm IR} = 10^{11-12}L_odot$; ULIRGs, $L_{rm IR}>10^{12}L_odot$). The hosts of the SMG ULIRGs, FIRG ULIRGs and FIRG LIRGs are of sufficient numbers to allow for detailed analysis, and they are only modestly different in their stellar masses. Their morphological types are predominantly disk galaxies (type D) and irregular/interacting systems (type Irr/Int). There is a morphological transition at $zapprox 1.25$ for the FIRG ULIRG hosts, above which the Irr/Int galaxies dominate and below which the D and the Irr/Int galaxies have nearly the same contributions. The SMG ULIRG hosts seem to experience a similar transition. This suggests a shift in the relative importance of galaxy mergers/interactions versus secular gas accretions in normal disk galaxies as the possible triggering mechanisms of ULIRGs. The FIRG LIRG hosts are predominantly D galaxies over $z=0.25-1.25$ where they are of sufficient statistics.