Our current understanding of radio-loud AGN comes predominantly from studies at frequencies of 5 GHz and below. With the recent completion of the Australia Telescope 20 GHz (AT20G) survey, we can now gain insight into the high-frequency radio properties of AGN. This paper presents supplementary information on the AT20G sources in the form of optical counterparts and redshifts. Optical counterparts were identified using the SuperCOSMOS database and redshifts were found from either the 6dF Galaxy survey or the literature. We also report 144 new redshifts. For AT20G sources outside the Galactic plane, 78.5% have optical identifications and 30.9% have redshift information. The optical identification rate also increases with increasing flux density. Targets which had optical spectra available were examined to obtain a spectral classification. There appear to be two distinct AT20G populations; the high luminosity quasars that are generally associated with point-source optical counterparts and exhibit strong emission lines in the optical spectrum, and the lower luminosity radio galaxies that are generally associated with passive galaxies in both the optical images and spectroscopic properties. It is suggested that these different populations can be associated with different accretion modes (cold-mode or hot-mode). We find that the cold-mode sources have a steeper spectral index and produce more luminous radio lobes, but generally reside in smaller host galaxies than their hot-mode counterparts. This can be attributed to the fact that they are accreting material more efficiently. Lastly, we compare the AT20G survey with the S-cubed semi-empirical (S3-SEX) models and conclude that the S3-SEX models need refining to correctly model the compact cores of AGN. The AT20G survey provides the ideal sample to do this.