Double electron capture is a rare nuclear decay process in which two orbital electrons are captured simultaneously in the same nucleus. Measurement of its two-neutrino mode would provide a new reference for the calculation of nuclear matrix elements whereas observation of its neutrinoless mode would demonstrate lepton number violation. A search for two-neutrino double electron capture on $^{124}$Xe is performed using 165.9 days of data collected with the XMASS-I liquid xenon detector. No significant excess above background was observed and we set a lower limit on the half-life as $4.7 times 10^{21}$ years at 90% confidence level. The obtained limit has ruled out parts of some theoretical expectations. We obtain a lower limit on the $^{126}$Xe two-neutrino double electron capture half-life of $4.3 times 10^{21}$ years at 90% confidence level as well.