The coupling between spin and charge degrees of freedom in a crystal imparts strong optical signatures on scattered electromagnetic waves. This has led to magneto-optical effects with a host of applications, from the sensitive detection of local magnetic order to optical modulation and data storage technologies. Here, we demonstrate a new magneto-optical effect, namely, the tuning of inelastically scattered light through symmetry control in atomically thin chromium triiodide (CrI$_3$). In monolayers, we found an extraordinarily large magneto-optical Raman effect from an A$_{1g}$ phonon mode due to the emergence of ferromagnetic order. The linearly polarized, inelastically scattered light rotates by ~40$^o$, more than two orders of magnitude larger than the rotation from MOKE under the same experimental conditions. In CrI$_3$ bilayers, we show that the same A$_{1g}$ phonon mode becomes Davydov-split into two modes of opposite parity, exhibiting divergent selection rules that depend on inversion symmetry and the underlying magnetic order. By switching between the antiferromagnetic states and the fully spin-polarized states with applied magnetic and electric fields, we demonstrate the magnetoelectrical control over their selection rules. Our work underscores the unique opportunities provided by 2D magnets for controlling the combined time-reversal and inversion symmetries to manipulate Raman optical selection rules and for exploring emergent magneto-optical effects and spin-phonon coupled physics.