The XENON100 experiment, installed underground at the Laboratori Nazionali del Gran Sasso (LNGS), aims to directly detect dark matter in the form of Weakly Interacting Massive Particles (WIMPs) via their elastic scattering off xenon nuclei. This paper presents a study on the nuclear recoil background of the experiment, taking into account neutron backgrounds from ($alpha$,n) and spontaneous fission reactions due to natural radioactivity in the detector and shield materials, as well as muon-induced neutrons. Based on Monte Carlo simulations and using measured radioactive contaminations of all detector components, we predict the nuclear recoil backgrounds for the WIMP search results published by the XENON100 experiment in 2011 and 2012, 0.11$^{+0.08}_{-0.04}$ events and 0.17$^{+0.12}_{-0.07}$ events, respectively, and conclude that they do not limit the sensitivity of the experiment.