X-shooter is one of the most popular instruments at the VLT, offering instantaneous spectroscopy from 300 to 2500 nm. We present the design of a single polarimetric unit at the polarization-free Cassegrain focus that serves all three spectrograph arms of X-shooter. It consists of a calcite Savart plate as a polarizing beam-splitter and a rotatable crystal retarder stack as a polychromatic modulator. Since even superachromatic wave plates have a wavelength range that is too limited for X-shooter, this novel modulator is designed to offer close-to-optimal polarimetric efficiencies for all Stokes parameters at all wavelengths. We analyze the modulator design in terms of its polarimetric performance, its temperature sensitivity, and its polarized fringes. Furthermore, we present the optical design of the polarimetric unit. The X-shooter polarimeter will furnish a myriad of science cases: from measuring stellar magnetic fields (e.g., Ap stars, white dwarfs, massive stars) to determining asymmetric structures around young stars and in supernova explosions.