We investigate the relation between AGN and star formation (SF) activity at $0.5 < z < 3$ by analyzing 898 galaxies with X-ray luminous AGN ($L_X > 10^{44}$ erg s$^{-1}$) and a large comparison sample of $sim 320,000$ galaxies without X-ray luminous AGN. Our samples are selected from a large (11.8 deg$^2$) area in Stripe 82 that has multi-wavelength (X-ray to far-IR) data. The enormous comoving volume ($sim 0.3$ Gpc$^3$) at $0.5 < z < 3$ minimizes the effects of cosmic variance and captures a large number of massive galaxies ($sim 30,000$ galaxies with $M_* > 10^{11} M_{odot}$) and X-ray luminous AGN. While many galaxy studies discard AGN hosts, we fit the SED of galaxies with and without X-ray luminous AGN with Code Investigating GALaxy Emission (CIGALE) and include AGN emission templates. We find that without this inclusion, stellar masses and star formation rates (SFRs) in AGN host galaxies can be overestimated, on average, by factors of up to $sim 5$ and $sim 10$, respectively. The average SFR of galaxies with X-ray luminous AGN is higher by a factor of $sim 3$ to $10$ compared to galaxies without X-ray luminous AGN at fixed stellar mass and redshift, suggesting that high SFRs and high AGN X-ray luminosities may be fueled by common mechanisms. The vast majority ($> 95 %$) of galaxies with X-ray luminous AGN at $z=0.5-3$ do not show quenched SF: this suggests that if AGN feedback quenches SF, the associated quenching process takes a significant time to act and the quenched phase sets in after the highly luminous phases of AGN activity.