In August 2017, the first detection of a binary neutron star merger, GW170817, made it possible to study neutron stars in compact binary systems using gravitational waves. Despite being the loudest (in terms of signal-to-noise ratio) gravitational wave detected to date, it was not possible to unequivocally determine that GW170817 was caused by the merger of two neutron stars instead of two black holes from the gravitational-wave data alone. That distinction was largely due to the accompanying electromagnetic counterpart. This raises the question: under what circumstances can gravitational-wave data alone, in the absence of an electromagnetic signal, be used to distinguish between different types of mergers? Here, we study whether a neutron-star--black-hole binary merger can be distinguished from a binary black hole merger using gravitational-wave data alone. We build on earlier results using chiral effective field theory to explore whether the data from LIGO and Virgo, LIGO A+, LIGO Voyager, or Cosmic Explorer could lead to such a distinction. The results suggest that the present LIGO-Virgo detector network will most likely be unable to distinguish between these systems even with the planned near-term upgrades. However, given an event with favorable parameters, third-generation instruments such as Cosmic Explorer will be capable of making this distinction. This result further strengthens the science case for third-generation detectors.