We present the results of deep spectropolarimetry of two powerful radio galaxies at $zsim2.5$ (4C 00.54 and 4C 23.56) obtained with the W.M. Keck II 10m telescope, aimed at studying the relative contribution of the stellar and non-stellar components to the ultraviolet continuum. Both galaxies show strong linear polarization of the continuum between rest-frame $sim$1300-2000~AA, and the orientation of the electric vector is perpendicular to the main axis of the UV continuum. In this sense, our objects are like most 3C radio galaxies at $zsim1$. The total flux spectra of 4C 00.54 and 4C 23.56 do not show the strong P-Cygni absorption features or the photospheric absorption lines expected when the UV continuum is dominated by young and massive stars. The only features detected can be ascribed to interstellar absorptions by SiII, CII and OI. Our results are similar to those for 3C radio galaxies at lower $z$, suggesting that the UV continuum of powerful radio galaxies at $zsim2.5$ is still dominated by non-stellar radiation, and that young massive stars do not contribute more than $approx$50% to the total continuum flux at 1500~AA.