Spin waves in chiral magnetic materials are strongly influenced by the Dzyaloshinskii-Moriya interaction resulting in intriguing phenomena like non-reciprocal magnon propagation and magnetochiral dichroism. Here, we study the non-reciprocal magnon spectrum of the archetypical chiral magnet MnSi and its evolution as a function of magnetic field covering the field-polarized and conical helix phase. Using inelastic neutron scattering, the magnon energies and their spectral weights are determined quantitatively after deconvolution with the instrumental resolution. In the field-polarized phase the imaginary part of the dynamical susceptibility $chi(varepsilon, {bf q})$ is shown to be asymmetric with respect to wavevectors ${bf q}$ longitudinal to the applied magnetic field ${bf H}$, which is a hallmark of chiral magnetism. In the helimagnetic phase, $chi(varepsilon, {bf q})$ becomes increasingly symmetric with decreasing ${bf H}$ due to the formation of helimagnon bands and the activation of additional spinflip and non-spinflip scattering channels. The neutron spectra are in excellent quantitative agreement with the low-energy theory of cubic chiral magnets with a single fitting parameter being the damping rate of spin waves.