The stellar initial mass function (IMF), which is often assumed to be universal across unresolved stellar populations, has recently been suggested to be bottom-heavy for massive ellipticals. In these galaxies, the prevalence of gravity-sensitive absorption lines (e.g. Na I and Ca II) in their near-IR spectra implies an excess of low-mass ($m <= 0.5$ $M_odot$) stars over that expected from a canonical IMF observed in low-mass ellipticals. A direct extrapolation of such a bottom-heavy IMF to high stellar masses ($m >= 8$ $M_odot$) would lead to a corresponding deficit of neutron stars and black holes, and therefore of low-mass X-ray binaries (LMXBs), per unit near-IR luminosity in these galaxies. Peacock et al. (2014) searched for evidence of this trend and found that the observed number of LMXBs per unit $K$-band luminosity ($N/L_K$) was nearly constant. We extend this work using new and archival Chandra X-ray Observatory (Chandra) and Hubble Space Telescope (HST) observations of seven low-mass ellipticals where $N/L_K$ is expected to be the largest and compare these data with a variety of IMF models to test which are consistent with the observed $N/L_K$. We reproduce the result of Peacock et al. (2014), strengthening the constraint that the slope of the IMF at $m >= 8$ $M_odot$ must be consistent with a Kroupa-like IMF. We construct an IMF model that is a linear combination of a Milky Way-like IMF and a broken power-law IMF, with a steep slope ($alpha_1=$ $3.84$) for stars < 0.5 $M_odot$ (as suggested by near-IR indices), and that flattens out ($alpha_2=$ $2.14$) for stars > 0.5 $M_odot$, and discuss its wider ramifications and limitations.