The origin of the astrophysical neutrino flux reported by the IceCube Collaboration remains an open question. In this study, we use three years of publicly available IceCube data to search for evidence of neutrino emission from the blazars and non-blazar Active Galactic Nuclei (AGN) contained the Fermi 4LAC catalog. We find no evidence that these sources produce high-energy neutrinos, and conclude that blazars can produce no more than 15% of IceCubes observed flux. The constraint we derive on the contribution from non-blazar AGN, which are less luminous and more numerous than blazars, is significantly less restrictive, and it remains possible that this class of sources could produce the entirety of the diffuse neutrino flux observed by IceCube. We anticipate that it will become possible to definitively test such scenarios as IceCube accumulates and releases more data, and as gamma-ray catalogs of AGN become increasingly complete. We also comment on starburst and other starforming galaxies, and conclude that these sources could contribute substantially to the signal observed by IceCube, in particular at the lowest detected energies.