We analyze the properly normalized three-point correlator of two protected scalar operators and one higher spin twist-two operator in N=4 super Yang-Mills, in the limit of large spin j. The relevant structure constant can be extracted from the OPE of the four-point correlator of protected scalar operators. We show that crossing symmetry of the four point correlator plus a judicious guess for the perturbative structure of the three-point correlator, allow to make a prediction for the structure constant at all loops in perturbation theory, up to terms that remain finite as the spin becomes large. Furthermore, the expression for the structure constant allows to propose an expression for the all loops four-point correlator G(u,v), in the limit u,v -> 0. Our predictions are in perfect agreement with the large j expansion of results available in the literature.