The X-ray emission from normal elliptical galaxies has two major components: soft emission from diffuse gas and harder emission from populations of accreting (low-mass) stellar X-ray binaries (LMXB). If LMXB populations are tied to the field stellar populations in galaxies, their total X-ray luminosities should be proportional to the optical luminosities of galaxies. However, recent ASCA and Chandra X-ray observations show that the global luminosities of LMXB components in ellipticals exhibit significant scatter at a given optical luminosity. This scatter may reflect a range of evolutionary stages among LMXB populations in ellipticals of different ages. If so, the ratio of the global LMXB X-ray luminosity to the galactic optical luminosity, L_LMXB/L_opt, may be used to determine when the bulk of stars were formed in individual ellipticals. To test this, we compare variations in L_LMXB/L_opt for LMXB populations in ellipticals to optically-derived estimates of stellar ages in the same galaxies. We find no correlation, implying that L_LMXB/L_opt variations are not good age indicators for ellipticals. Alternatively, LMXBs may be formed primarily in globular clusters (through stellar tidal interactions), rather than in the stellar fields of galaxies. Since elliptical galaxies exhibit a wide range of globular cluster populations for a given galaxian luminosity, this may induce a dispersion in the LMXB populations of ellipticals with similar optical luminosities. Indeed, we find that L_LMXB/L_opt ratios for LMXB populations are strongly correlated with the specific globular cluster frequencies in elliptical galaxies. This suggests that most LMXBs were formed in globular clusters.