Active galactic nuclei (AGN) with jets seen at small viewing angles are the most luminous and abundant objects in the $gamma$-ray sky. AGN with jets misaligned along the line-of-sight appear fainter in the sky, but are more numerous than the brighter blazars. We calculate the diffuse $gamma$-ray emission due to the population of misaligned AGN (MAGN) unresolved by the Large Area Telescope (LAT) on the {it Fermi} Gamma-ray Space Telescope ({it Fermi}). A correlation between the $gamma$-ray luminosity and the radio-core luminosity is established and demonstrated to be physical by statistical tests, as well as compatible with upper limits based on {it Fermi}-LAT data for a large sample of radio-loud MAGN. We constrain the derived $gamma$-ray luminosity function by means of the source count distribution of the radio galaxies (RGs) detected by the {it Fermi}-LAT. We finally calculate the diffuse $gamma$-ray flux due to the whole MAGN population. Our results demonstrate that the MAGN can contribute from 10% up to nearly the entire measured Isotropic Gamma-Ray Background (IGRB). We evaluate a theoretical uncertainty on the flux of almost an order of magnitude.