Many questions are still open regarding the physical mechanisms behind the magnetic switching in GdFeCo alloys by single optical pulses. Phenomenological models suggest a femtosecond scale exchange relaxation between sublattice magnetization as the driving mechanism for switching. The recent observation of thermally induced switching in GdFeCo by using both several picosecond optical laser pulse as well as electric current pulses has questioned this previous understanding. This has raised the question of whether or not the same switching mechanics are acting at the femo- and picosecond scales. In this work, we aim at filling this gap in the understanding of the switching mechanisms behind thermal single-pulse switching. To that end, we have studied experimentally thermal single-pulse switching in GdFeCo alloys, for a wide range of system parameters, such as composition, laser power and pulse duration. We provide a quantitative description of the switching dynamics using atomistic spin dynamics methods with excellent agreement between the model and our experiments across a wide range of parameters and timescales, ranging from femtoseconds to picoseconds. Furthermore, we find distinct element-specific damping parameters as a key ingredient for switching with long picosecond pulses and argue, that switching with pulse durations as long as 15 picoseconds is possible due to a low damping constant of Gd. Our findings can be easily extended to speed up dynamics in other contexts where ferrimagnetic GdFeCo alloys have been already demonstrated to show fast and energy-efficient processes, e.g. domain-wall motion in a track and spin-orbit torque switching in spintronics devices.