CMOS Monolithic Active Pixel Sensors (MAPS) were chosen as sensor technology for the vertex detectors of STAR, CBM and the upgraded ALICE-ITS. They also constitute a valuable option for tracking devices at future e+e- colliders. Those applications require a substantial tolerance to both, ionizing and non-ionizing radiation. To allow for a focused optimization of the radiation tolerance, prototypes are tested by irradiating the devices either with purely ionizing radiation (e.g. soft X-rays) or the most pure sources of non-ionizing radiation available (e.g. reactor neutrons). In the second case, it is typically assumed that the impact of the parasitic $gamma$-rays found in the neutron beams is negligible. We checked this assumption by irradiating MAPS with $gamma$-rays and comparing the radiation damage generated with the one in neutron irradiated sensors. We conclude that the parasitic radiation doses may cause non-negligible radiation damage. Based on the results we propose a procedure to recognize and to suppress the effect of the related parasitic ionizing radiation damage.