Cosmic-ray protons accumulate for cosmological times in clusters of galaxies as their typical radiative and diffusive escape times are longer than the Hubble time. Their hadronic interactions with protons of the intra-cluster medium generate secondary electrons, gamma-rays and neutrinos. We here estimate the contribution from clusters to the diffuse gamma-ray and neutrino backgrounds. We model the cluster population by means of their mass function, using a phenomenological luminosity-mass relation applied to all clusters, as well as a detailed semi-analytical model. Additionally, we consider observationally-motivated values for the cluster magnetic field. This is a crucial parameter since the observed radio counts due to synchrotron emission by secondary electrons need to be respected. For a choice of parameters respecting all current constraints, and assuming a spectral index of -2, we find that hadronic interactions in clusters contribute by less than 10% to the extragalactic gamma-ray background observed by Fermi and to the IceCube flux. They account for less than 1% for spectral indices <=-2. The neutrino flux observed by IceCube can be reproduced without violating constraints only if a very hard (and speculative) spectral index >-2 is adopted. However, this scenario is in tension with the IceCube data, which seem to suggest a spectral energy distribution of the neutrino flux that decreases with energy. In the case of proton-photon interactions, we find that very likely protons do not reach sufficiently high energies to produce neutrinos in clusters. We argue that our results are optimistic due to our assumptions, and that clusters of galaxies cannot give any relevant contribution to the extragalactic gamma-ray and neutrino backgrounds. Finally, we find that the cluster contribution to the angular fluctuations in the gamma-ray background is subdominant, less than 10%. [abridged]