Primordial magnetic fields lead to non-Gaussian signals in the cosmic microwave background (CMB) even at the lowest order, as magnetic stresses and the temperature anisotropy they induce depend quadratically on the magnetic field. In contrast, CMB non-Gaussianity due to inflationary scalar perturbations arises only as a higher order effect. Apart from a compensated scalar mode, stochastic primordial magnetic fields also produce scalar anisotropic stress that remains uncompensated till neutrino decoupling. This gives rise to an adiabatic-like scalar perturbation mode that evolves passively thereafter (called the passive mode). We compute the CMB reduced bispectrum ($b_{l_{_1}l_{_2}l_{_3}}$) induced by this passive mode, sourced via the Sachs-Wolfe effect, on large angular scales. For any configuration of bispectrum, taking a partial sum over mode-coupling terms, we find a typical value of $l_1(l_1+1)l_3(l_3+1) b_{l_{_1}l_{_2}l_{_3}} sim 6-9 times 10^{-16}$, for a magnetic field of $B_0 sim 3$ nG, assuming a nearly scale-invariant magnetic spectrum . We also evaluate, in full, the bispectrum for the squeezed collinear configuration over all angular mode-coupling terms and find $l_1(l_1+1)l_3(l_3+1) b_{l_{_1}l_{_2}l_{_3}} approx -1.4 times 10^{-16}$. These values are more than $sim 10^6$ times larger than the previously calculated magnetic compensated scalar mode CMB bispectrum. Observational limits on the bispectrum from WMAP7 data allow us to set upper limits of $B_0 sim 2$ nG on the present value of the cosmic magnetic field of primordial origin. This is over 10 times more stringent than earlier limits on $B_0$ based on the compensated mode bispectrum.