We discuss the manner in which the primordial magnetic field (PMF) suppresses the cosmic microwave background (CMB) $B$ mode due to the weak-lensing (WL) effect. The WL effect depends on the lensing potential (LP) caused by matter perturbations, the distribution of which at cosmological scales is given by the matter power spectrum (MPS). Therefore, the WL effect on the CMB $B$ mode is affected by the MPS. Considering the effect of the ensemble average energy density of the PMF, which we call the background PMF, on the MPS, the amplitude of MPS is suppressed in the wave number range of $k>0.01~h$ Mpc$^{-1}$.The MPS affects the LP and the WL effect in the CMB $B$ mode; however, the PMF can damp this effect. Previous studies of the CMB $B$ mode with the PMF have only considered the vector and tensor modes. These modes boost the CMB $B$ mode in the multipole range of $ell > 1000$, whereas the background PMF damps the CMB $B$ mode owing to the WL effect in the entire multipole range. The matter density in the Universe controls the WL effect. Therefore, when we constrain the PMF and the matter density parameters from cosmological observational data sets, including the CMB $B$ mode, we expect degeneracy between these parameters. The CMB $B$ mode also provides important information on the background gravitational waves, inflation theory, matter density fluctuations, and the structure formations at the cosmological scale through the cosmological parameter search. If we study these topics and correctly constrain the cosmological parameters from cosmological observations including the CMB $B$ mode, we need to correctly consider the background PMF.