Cosmic rays are the highest energy particles found in nature. Measurements of the mass composition of cosmic rays between 10^{17} eV and 10^{18} eV are essential to understand whether this energy range is dominated by Galactic or extragalactic sources. It has also been proposed that the astrophysical neutrino signal comes from accelerators capable of producing cosmic rays of these energies. Cosmic rays initiate cascades of secondary particles (air showers) in the atmosphere and their masses are inferred from measurements of the atmospheric depth of the shower maximum, Xmax, or the composition of shower particles reaching the ground. Current measurements suffer from either low precision, or a low duty cycle and a high energy threshold. Radio detection of cosmic rays is a rapidly developing technique, suitable for determination of Xmax with a duty cycle of in principle nearly 100%. The radiation is generated by the separation of relativistic charged particles in the geomagnetic field and a negative charge excess in the shower front. Here we report radio measurements of Xmax with a mean precision of 16 g/cm^2 between 10^{17}-10^{17.5} eV. Because of the high resolution in $Xmax we can determine the mass spectrum and find a mixed composition, containing a light mass fraction of ~80%. Unless the extragalactic component becomes significant already below 10^{17.5} eV, our measurements indicate an additional Galactic component dominating at this energy range.