We investigate the impact of cosmic rays (CRs) on the circumgalactic medium (CGM) in FIRE-2 simulations, for ultra-faint dwarf through Milky Way (MW)-mass halos hosting star-forming (SF) galaxies. Our CR treatment includes injection by supernovae, anisotropic streaming and diffusion along magnetic field lines, collisional and streaming losses, with constant parallel diffusivity $kappasim3times10^{29},mathrm{cm^2 s^{-1}}$ chosen to match $gamma$-ray observations. With this, CRs become more important at larger halo masses and lower redshifts, and dominate the pressure in the CGM in MW-mass halos at $zlesssim 1-2$. The gas in these CR-dominated halos differs significantly from runs without CRs: the gas is primarily cool (a few $sim10^{4},$K), and the cool phase is volume-filling and has a thermal pressure below that needed for virial or local thermal pressure balance. Ionization of the low and mid ions in this diffuse cool gas is dominated by photo-ionization, with O VI columns $gtrsim 10^{14.5},mathrm{cm^{-2}}$ at distances $gtrsim 150,mathrm{kpc}$. CR and thermal gas pressure are locally anti-correlated, maintaining total pressure balance, and the CGM gas density profile is determined by the balance of CR pressure gradients and gravity. Neglecting CRs, the same halos are primarily warm/hot ($Tgtrsim 10^{5},$K) with thermal pressure balancing gravity, collisional ionization dominates, O VI columns are lower and Ne VIII higher, and the cool phase is confined to dense filaments in local thermal pressure equilibrium with the hot phase.