We study the large scale geometry of the relative free splitting complex and the relative complex of free factor systems of the rank $n$ free group $F_n$, relative to the choice of a free factor system of $F_n$, proving that these complexes are hyperbolic. Furthermore we present the proof in a general context, obtaining hyperbolicity of the relative free splitting complex and of the relative complex of free factor systems of a general group $Gamma$, relative to the choice of a free factor system of $Gamma$. The proof yields information about coarsely transitive families of quasigeodesics in each of these complexes, expressed in terms of fold paths of free splittings.