During the first few hundred days after the explosion, core-collapse supernovae (SNe) emit down-scattered X-rays and gamma-rays originating from radioactive line emissions, primarily from the $^{56}$Ni $rightarrow$ $^{56}$Co $rightarrow$ $^{56}$Fe chain. We use SN models based on three-dimensional neutrino-driven explosion simulations of single stars and mergers to compute this emission and compare the predictions with observations of SN 1987A. A number of models are clearly excluded, showing that high-energy emission is a powerful way of discriminating between models. The best models are almost consistent with the observations, but differences that cannot be matched by a suitable choice of viewing angle are evident. Therefore, our self-consistent models suggest that neutrino-driven explosions are able to produce, in principle, sufficient mixing, although remaining discrepancies may require small changes to the progenitor structures. The soft X-ray cutoff is primarily determined by the metallicity of the progenitor envelope. The main effect of asymmetries is to vary the flux level by a factor of ${sim}$3. For the more asymmetric models, the shapes of the light curves also change. In addition to the models of SN 1987A, we investigate two models of Type II-P SNe and one model of a stripped-envelope Type IIb SN. The Type II-P models have similar observables as the models of SN 1987A, but the stripped-envelope SN model is significantly more luminous and evolves faster. Finally, we make simple predictions for future observations of nearby SNe.