Lattice QCD simulations are now reaching a precision where isospin breaking effects become important. Previously, we have developed a program to systematically investigate the pattern of flavor symmetry beaking within QCD and successfully applied it to meson and baryon masses involving up, down and strange quarks. In this Letter we extend the calculations to QCD + QED and present our first results on isospin splittings in the pseudoscalar meson and baryon octets. In particular, we obtain the nucleon mass difference of $M_n-M_p=1.35(18)(8),mbox{MeV}$ and the electromagnetic contribution to the pion splitting $M_{pi^+}-M_{pi^0}=4.60(20),mbox{MeV}$. Further we report first determination of the separation between strong and electromagnetic contributions in the $bar{MS}$ scheme.