Observations of both galaxies in the distant Universe and local starbursts are showing increasing evidence for very hard ionizing spectra that stellar population synthesis models struggle to reproduce. Here we explore the effects of the assumed stellar initial mass function (IMF) on the ionizing photon output of young populations at wavelengths below key ionization energy thresholds. We use a custom set of binary population and spectral synthesis (BPASS) models to explore the effects of IMF assumptions as a function of metallicity, IMF slope, upper mass limit, IMF power law break mass and sampling. We find that while the flux capable of ionizing hydrogen is only weakly dependent on IMF parameters, the photon flux responsible for the He II and O VI lines is far more sensitive to assumptions. In our current models this flux arises primarily from helium and Wolf-Rayet stars which have partially or fully lost their hydrogen envelopes. The timescales for formation and evolution of both Wolf Rayet stars and helium dwarfs, and hence inferred population age, are affected by choice of model IMF. Even the most extreme IMFs cannot reproduce the He II ionizing flux observed in some high redshift galaxies, suggesting a source other than stellar photospheres. We caution that detailed interpretation of features in an individual galaxy spectrum is inevitably going to be subject to uncertainties in the IMF of its contributing starbursts. We remind the community that the initial mass function is fundamentally a statistical construct, and that stellar population synthesis models are most effective when considering entire galaxy populations rather than individual objects.