We investigate finite size effects of the pion matrix element of the non-singlet, twist-2 operator corresponding to the average momentum of non-singlet quark densities. Using the quenched approximation, they come out to be surprisingly large when compared to the finite size effects of the pion mass. As a consequence, simulations of corresponding nucleon matrix elements could be affected by finite size effects even stronger which could lead to serious systematic uncertainties in their evaluation.