Compared to traditional nonlinear optical crystals, like BaB$_2$O$_4$, KTiOPO$_4$ or LiNbO$_3$, semiconductor integrated sources of photon pairs may operate at pump wavelengths much closer to the bandgap of the materials. This is also the case for Bragg-reflection waveguides (BRW) targeting parametric down-conversion (PDC) to the telecom C-band. The large nonlinear coefficient of the AlGaAs alloy and the strong confinement of the light enable extremely bright integrated photon pair sources. However, under certain circumstances, a significant amount of detrimental broadband photoluminescence has been observed in BRWs. We show that this is mainly a result of linear absorption near the core and subsequent radiative recombination of electron-hole pairs at deep impurity levels in the semiconductor. For PDC with BRWs, we conclude that devices operating near the long wavelength end of the S-band or the short C-band require temporal filtering shorter than 1 ns. We predict that shifting the operating wavelengths to the L-band and making small adjustments in the material composition will reduce the amount of photoluminescence to negligible values. Such measures enable us to increase the average pump power and/or the repetition rate, which makes integrated photon pair sources with on-chip multi-gigahertz pair rates feasible.