We present the implementation and the first results of cosmic ray (CR) feedback in the Feedback In Realistic Environments (FIRE) simulations. We investigate CR feedback in non-cosmological simulations of dwarf, sub-$Lstar$ starburst, and $Lstar$ galaxies with different propagation models, including advection, isotropic and anisotropic diffusion, and streaming along field lines with different transport coefficients. We simulate CR diffusion and streaming simultaneously in galaxies with high resolution, using a two moment method. We forward-model and compare to observations of $gamma$-ray emission from nearby and starburst galaxies. We reproduce the $gamma$-ray observations of dwarf and $Lstar$ galaxies with constant isotropic diffusion coefficient $kappa sim 3times 10^{29},{rm cm^{2},s^{-1}}$. Advection-only and streaming-only models produce order-of-magnitude too large $gamma$-ray luminosities in dwarf and $Lstar$ galaxies. We show that in models that match the $gamma$-ray observations, most CRs escape low-gas-density galaxies (e.g. dwarfs) before significant collisional losses, while starburst galaxies are CR proton calorimeters. While adiabatic losses can be significant, they occur only after CRs escape galaxies, so they are only of secondary importance for $gamma$-ray emissivities. Models where CRs are ``trapped in the star-forming disk have lower star formation efficiency, but these models are ruled out by $gamma$-ray observations. For models with constant $kappa$ that match the $gamma$-ray observations, CRs form extended halos with scale heights of several kpc to several tens of kpc.