Context: Only two planetary systems around old ms-pulsars are currently known. Young radio pulsars and radio-quiet neutron stars cannot be covered by the usually-applied radio pulse timing technique. However, finding substellar companions around these neutron stars would be of great interest -- not only because of the companions possible exotic formation but also due to the potential access to neutron star physics. Aims: We investigate the closest young neutron stars to search for substellar companions around them. Methods: Young, thus warm substellar companions are visible in the Near Infrared while the neutron star itself is much fainter. Four young neutron stars are moving fast enough to enable a common proper motion search for substellar companions within few years. Results. For Geminga, RX J0720.4-3125, RX J1856.6-3754, and PSR J1932+1059 we did not find any co-moving companion down to 12, 15, 11, 42 Jupiter masses for assumed ages of 1, 1, 1, 3.1 Myrs and distances of 250, 361, 167, 361 pc, respectively. Near Infrared limits are presented for these four as well as five other neutron stars for which we currently have only observations at one epoch. Conclusions: We conclude that young isolated neutron stars rarely have brown dwarf companions.