Transition-metal interfaces and multilayers are a very promising class of systems to realize nanometer-sized, stable magnetic skyrmions for future spintronic devices. For room temperature applications it is crucial to understand the interactions which control the stability of isolated skyrmions. Typically, skyrmion properties are explained by the interplay of pair-wise exchange interactions, the Dzyaloshinskii-Moriya interaction and the magnetocrystalline anisotropy energy. Here, we demonstrate that higher-order exchange interactions -- which have so far been neglected -- can play a key role for the stability of skyrmions. We use an atomistic spin model parametrized from first-principles and compare three different ultrathin film systems. We consider all fourth order exchange interactions and show that in particular the four-site four spin interaction has a giant effect on the energy barrier preventing skyrmion and antiskyrmion collapse into the ferromagnetic state. Our work opens new perspectives to enhance the stability of topological spin structures.