The generalized seniority scheme has long been proposed as a means of dramatically reducing the dimensionality of nuclear shell model calculations, when strong pairing correlations are present. However, systematic benchmark calculations, comparing results obtained in a model space truncated according to generalized seniority with those obtained in the full shell model space, are required to assess the viability of this scheme. Here, a detailed comparison is carried out, for semimagic nuclei taken in a full major shell and with realistic interactions. The even-mass and odd-mass Ca isotopes are treated in the generalized seniority scheme, for generalized seniority v<=3. Results for level energies, orbital occupations, and electromagnetic observables are compared with those obtained in the full shell model space.