The J-PARC E56 experiment aims to search for sterile neutrinos at the J-PARC Materials and Life Science Experimental Facility (MLF). In order to examine the feasibility of the experiment, we measured the background rates of different detector candidate sites, which are located at the third floor of the MLF, using a detector consisting of plastic scintillators with a fiducial mass of 500 kg. The result of the measurements is described in this article. The gammas and neutrons induced by the beam as well as the backgrounds from the cosmic rays were measured.