We present a study of 15 long-duration gamma-ray burst (GRB) host galaxies at z>2. The GRBs are selected with available early-time afterglow spectra in order to compare interstellar medium (ISM) absorption-line properties with stellar properties of the host galaxies. In addition to five previously studied hosts, we consider new detections for the host galaxies of GRB050820 and GRB060206 and place 2-sigma upper limits to the luminosities of the remaining unidentified hosts. We examine the nature of the host galaxy population and find that (1) the UV luminosity distribution of GRB host galaxies is consistent with expectations from a UV luminosity weighted random galaxy population with a median luminosity of <L(UV)>=0.1 L*; (2) there exists a moderate correlation between UV luminosity and SiII 1526 absorption width, which together with the observed large line widths of W(1526)>1.5 Ang for a large fraction of the objects suggests a galactic outflow driven velocity field in the host galaxies; (3) there is tentative evidence for a trend of declining ISM metallicity with decreasing galaxy luminosity in the star-forming galaxy population at z=2-4; (4) the interstellar UV radiation field is found ~ 35-350 times higher in GRB hosts than the Galactic mean value; and (5) additional galaxies are found at < 2 from the GRB host in all fields with known presence of strong MgII absorbers, but no additional faint galaxies are found at < 2 in fields without strong MgII absorbers. Our study confirms that the GRB host galaxies (with known optical afterglows) are representative of unobscured star-forming galaxies at z>2, and demonstrates that high spatial resolution images are necessary for an accurate identification of GRB host galaxies in the presence of strong intervening absorbers.