We investigate the stellar and dust properties of massive (log$(M_*/M_odot) ge 10.5$) and dusty ($A_V ge 1$) galaxies at $1 le z le 4$ by modeling their spectral energy distributions (SEDs) obtained from the combination of UltraVISTA DR3 photometry and textit{Herschel} PACS-SPIRE data using MAGPHYS. Although the rest-frame U-V vs V-J (UVJ) diagram traces well the star-formation rates (SFR) and dust obscuration (A$_V$) out to $z sim 3$, $sim$15-20% of the sample surprisingly resides in the quiescent region of the UVJ diagram, while $sim50$% at $3<z<4$ fall in the unobscured star-forming region. The median SED of massive dusty galaxies exhibits weaker MIR and UV emission, and redder UV slopes with increasing cosmic time. The IR emission for our sample has a significant contribution ($>20%$) from dust heated by evolved stellar populations rather than star formation, demonstrating the need for panchromatic SED modeling. The local relation between dust mass and SFR is followed only by a sub-sample with cooler dust temperatures, while warmer objects have reduced dust masses at a given SFR. Most star-forming galaxies in our sample do not follow local IRX-$beta$ relations, though IRX does strongly correlate with A$_V$. Our sample follows local relations, albeit with large scatter, between ISM diagnostics and sSFR. We show that FIR-detected sources represent the extreme of a continuous population of dusty galaxies rather than a fundamentally different population. Finally, using commonly adopted relations to derive SFRs from the combination of the rest-frame UV and the observed 24$mu$m is found to overestimate the SFR by a factor of 3-5 for the galaxies in our sample.