We determine the generalized form factors, which correspond to the second Mellin moment (i.e., the first $x$-moment) of the generalized parton distributions of the nucleon at leading twist. The results are obtained using lattice QCD with $N_f=2$ nonperturbatively improved Wilson fermions, employing a range of quark masses down to an almost physical value with a pion mass of about 150 MeV. We also present results for the isovector quark angular momentum and for the first $x$-moment of the transverse quark spin density. We compare two different fit strategies and find that directly fitting the ground state matrix elements to the functional form expected from Lorentz invariance and parametrized in terms of form factors yields comparable, and usually more stable results than the traditional approach where the form factors are determined from an overdetermined linear system based on the fitted matrix elements.