Ultralight bosons can form large clouds around stellar-mass black holes via the superradiance instability. Through processes such as annihilation, these bosons can source continuous gravitational wave signals with frequencies within the range of LIGO and Virgo. If boson annihilation occurs, then the Galactic black hole population will give rise to many gravitational signals; we refer to this as the ensemble signal. We characterize the ensemble signal as observed by the gravitational-wave detectors; this is important because the ensemble signal carries the primary signature that a continuous wave signal has a boson annihilation origin. We explore how a broad set of black hole population parameters affects the resulting spin-0 boson annihilation signal and consider its detectability by recent searches for continuous gravitational waves. A population of $10^8$ black holes with masses up to $30mathrm{M}_odot$ and a flat dimensionless initial spin distribution between zero and unity produces up to a thousand signals loud enough to be in principle detected by these searches. For a more moderately spinning population the number of signals drops by about an order of magnitude, still yielding up to a hundred detectable signals for some boson masses. A non-detection of annihilation signals at frequencies between 100 and 1200 Hz disfavors the existence of scalar bosons with rest energies between $2times10^{-13}$ and $2.5times10^{-12}$ eV. Finally we show that, depending on the black hole population parameters, care must be taken in assuming that the continuous wave upper limits from searches for isolated signals are still valid for signals that are part of a dense ensemble: Between 200 and 300 Hz, we urge caution when interpreting a null result for bosons between 4 and $6times10^{-13}$ eV.