We have uncovered a giant gyrotropic magneto-optical response for doped ferromagnetic manganite La2/3Ca1/3MnO3 around the near room-temperature paramagnetic-to-ferromagnetic transition. At odds with current wisdom, where this response is usually assumed to be fundamentally fixed by the electronic band structure, we point to the presence of small polarons as the driving force for this unexpected phenomenon. We explain the observed properties by the intricate interplay of mobility, Jahn-Teller effect and spin-orbit coupling of small polarons. As magnetic polarons are ubiquitously inherent to many strongly correlated systems, our results provide an original, general pathway towards the generation of gigantic gyrotropic responses that can be harnessed for nonreciprocal devices that exploit the polarization of light.