We investigate the validity of the quasar - radio galaxy unification scenario and detect dust tori within radio galaxies of various types. Using VISIR on the VLT, we acquired sub-arcsecond (~0.40) resolution N-band images, at a wavelength of 11.85 micron, of the nuclei of a sample of 27 radio galaxies of four types in the redshift range z=0.006-0.156. The sample consists of 8 edge-darkened, low-power Fanaroff-Riley class I (FR-I) radio galaxies, 6 edge-brightened, class II (FR-II) radio galaxies displaying low-excitation optical emission, 7 FR-IIs displaying high-excitation optical emission, and 6 FR-II broad emission line radio galaxies. Out of the sample of 27 objects, 10 nuclei are detected and several have constraining non-detections at sensitivities of 7 mJy, the limiting flux a point source has when detected with a signal-to-noise ratio of 10 in one hour of source integration. On the basis of the core spectral energy distributions of this sample we find clear indications that many FR-I and several low-excitation FR-II radio galaxies do not contain warm dust tori. At least 57+-19 percent of the high-excitation FR-IIs and almost all broad line radio galaxies display excess infrared emission, which must be attributed to warm dust reradiating accretion activity. The FR-I and low-excitation FR-II galaxies all possess low efficiencies, calculated as the ratio of bolometric and Eddington luminosity log (L_bol/L_Edd) < -3. This suggests that thick tori are absent at low accretion rates and/or low efficiencies. We argue that the unification viewing angle range 0-45 degrees of quasars should be increased to ~60 degrees, at least at lower luminosities.