The rest-frame UV emission from massive stars contains a wealth of information about the physical nature and conditions of star formation in galaxies. Using studies of the rest-frame UV, the past decade has witnessed the beginning of knowledge about the existence and properties of galaxies during the first few billion years after the Big Bang. This period of history corresponds to the formation of the first stars, the rapid formation of galaxy stellar populations, the reionization of the IGM, the production and dissemination of heavy elements, and the formation of the first black holes. Massive stars in these galaxies drive all of these events, and their light dominates the spectral energy distributions of galaxies. As we look to the 2020s, fundamental questions remain about the nature of these stellar populations and their evolution, from just before the peak of the cosmic star formation density (z~3), up to the epoch of reionization (z > 6). This next decade will provide transformative gains both in our ability to identify star-forming galaxies and accreting supermassive black holes at these early epochs with imaging surveys in the rest-frame UV (e.g., LSST, WFIRST). Ground-based, rest-frame UV spectroscopy on >20 m-class telescopes (e.g., GMT/TMT) offers the ability to investigate the astrophysical conditions in galaxies at the earliest cosmic times. This includes studies of the evolution in galaxy stellar populations, gas ionization (temperature, pressure), metallicity, and interstellar (and circumgalactic) gas kinematics and covering fractions. In this white paper, we describe the scientific prospects and the requirements for research in this area.