We developed a fast X-ray detector system for nuclear resonant scattering (NRS) experiments. Our system employs silicon avalanche photo-diode (Si-APD) as a fast X-ray sensor. The system is able to acquire both timing and energy of a single X-ray photon simultaneously in a high rate condition, $10^6$ counts per second for one Si-APD. The performance of the system was investigated in SPring-8, a synchrotron radiation facility in Japan. Good time resolution of SI{120}{ps} (FWHM) was achieved with a slight tail distribution in the time spectrum by a level of $10^{-9}$ at 1 ns apart from the peak. Using this system, we successfully observed the NRS from the 26.27-keV level of mercury-201, which has a half-life of $630 pm 50$ ps. We also demonstrated the reduction of background events caused by radioactive decays in a radioactive sample by discriminating photon energy.