We argue that the centrosymmetric $C2/c$ symmetry in BiMnO$_3$ is spontaneously broken by antiferromagnetic (AFM) interactions existing in the system. The true symmetry is expected to be $Cc$, which is compatible with the noncollinear magnetic ground state, where the ferromagnetic order along one crystallographic axis coexists with the the hidden AFM order and related to it ferroelectric polarization along two other axes. The $C2/c$ symmetry can be restored by the magnetic field $B sim 35$ Tesla, which switches off the ferroelectric polarization. Our analysis is based on the solution of the low-energy model constructed for the 3d-bands of BiMnO$_3$, where all the parameters have been derived from the first-principles calculations. Test calculations for isostructural BiCrO$_3$ reveal an excellent agreement with experimental data.