Supernova fallback disks around neutron stars have been discussed to influence the evolution of the diverse neutron star populations. Slowly rotating neutron stars are most promising to find such disks. Searching for the cold and warm debris of old fallback disks, we carried out Herschel PACS (70 $mu$m, 160 $mu$m) and Spitzer IRAC (3.6 $mu$m, 4.5 $mu$m) observations of eight slowly rotating ($Papprox 3 - 11$ s) nearby ($<1$ kpc) isolated neutron stars. Herschel detected 160 $mu$m emission ($>5sigma$) at locations consistent with the positions of the neutron stars RX J0806.4-4123 and RX J2143.0+0654. No other significant infrared emission was detected from the eight neutron stars. We estimate probabilities of 63%, 33% and 3% that, respectively, none, one, or both Herschel PACS 160 $mu$m detections are unrelated excess sources due to background source confusion or an interstellar cirrus. If the 160 $mu$m emission is indeed related to cold (10 K to 22 K) dust around the neutron stars, this dust is absorbing and re-emitting $sim 10$% to $sim 20$% of the neutron stars X-rays. Such high efficiencies would be at least three orders of magnitude larger than the efficiencies of debris disks around nondegenerate stars. While thin dusty disks around the neutron stars can be excluded as counterparts of the 160 $mu$m emission, dusty asteroid belts constitute a viable option.