We present the Empirical Dust Attenuation (EDA) framework -- a flexible prescription for assigning realistic dust attenuation to simulated galaxies based on their physical properties. We use the EDA to forward model synthetic observations for three state-of-the-art large-scale cosmological hydrodynamical simulations: SIMBA, IllustrisTNG, and EAGLE. We then compare the optical and UV color-magnitude relations, $(g-r) - M_r$ and $(FUV-NUV)-M_r$, of the simulations to a $M_r < -20$ and UV complete SDSS galaxy sample using likelihood-free inference. Without dust, none of the simulations match observations, as expected. With the EDA, however, we can reproduce the observed color-magnitude with all three simulations. Furthermore, the attenuation curves predicted by our dust prescription are in good agreement with the observed attenuation-slope relations and attenuation curves of star-forming galaxies. However, the EDA does not predict star-forming galaxies with low $A_V$ since simulated star-forming galaxies are intrinsically much brighter than observations. Additionally, the EDA provides, for the first time, predictions on the attenuation curves of quiescent galaxies, which are challenging to measure observationally. Simulated quiescent galaxies require shallower attenuation curves with lower amplitude than star-forming galaxies. The EDA, combined with forward modeling, provides an effective approach for shedding light on dust in galaxies and probing hydrodynamical simulations. This work also illustrates a major limitation in comparing galaxy formation models: by adjusting dust attenuation, simulations that predict significantly different galaxy populations can reproduce the same UV and optical observations.