We examine the late-time (t > 200 days after peak brightness) spectra of Type Iax supernovae (SNe Iax), a low-luminosity, low-energy class of thermonuclear stellar explosions observationally similar to, but distinct from, Type Ia supernovae. We present new spectra of SN 2014dt, resulting in the most complete published late-time spectral sequence of a SN Iax. At late times, SNe Iax have generally similar spectra, all with a similar continuum shape and strong forbidden-line emission. However, there is also significant diversity where some late-time SN Iax spectra display narrow P-Cygni features and a continuum indicative of a photosphere in addition to strong narrow forbidden lines, while others have no obvious P-Cygni features, strong broad forbidden lines, and weak narrow forbidden lines. Finally, some SNe Iax have spectra intermediate to these two varieties with weak P-Cygni features and broad/narrow forbidden lines of similar strength. We find that SNe Iax with strong broad forbidden lines also tend to be more luminous and have higher-velocity ejecta at peak brightness. We estimate blackbody and kinematic radii of the late-time photosphere, finding the latter an order of magnitude larger than the former. We propose a two-component model that solves this discrepancy and explains the diversity of the late-time spectra of SNe Iax. In this model, the broad forbidden lines originate from the SN ejecta, while the photosphere, P-Cygni lines, and narrow forbidden lines originate from a wind launched from the remnant of the progenitor white dwarf and is driven by the radioactive decay of newly synthesized material left in the remnant. The relative strength of the two components accounts for the diversity of late-time SN Iax spectra. This model also solves the puzzle of a long-lived photosphere and slow late-time decline of SNe Iax. (Abridged)