Context. Circumstellar disks and self-luminous giant exoplanets or companion brown dwarfs can be characterized through direct-imaging polarimetry at near-infrared wavelengths. SPHERE/IRDIS at the Very Large Telescope has the capabilities to perform such measurements, but uncalibrated instrumental polarization effects limit the attainable polarimetric accuracy. Aims. We aim to characterize and correct the instrumental polarization effects of the complete optical system, i.e. the telescope and SPHERE/IRDIS. Methods. We create a detailed Mueller matrix model in the broadband filters Y-, J-, H- and Ks, and calibrate it using measurements with SPHEREs internal light source and observations of two unpolarized stars. We develop a data-reduction method that uses the model to correct for the instrumental polarization effects, and apply it to observations of the circumstellar disk of T Cha. Results. The instrumental polarization is almost exclusively produced by the telescope and SPHEREs first mirror and varies with telescope altitude angle. The crosstalk primarily originates from the image derotator (K-mirror). At some orientations, the derotator causes severe loss of signal (>90% loss in H- and Ks-band) and strongly offsets the angle of linear polarization. With our correction method we reach in all filters a total polarimetric accuracy of <0.1% in the degree of linear polarization and an accuracy of a few degrees in angle of linear polarization. Conclusions. The correction method enables us to accurately measure the polarized intensity and angle of linear polarization of circumstellar disks, and is a vital tool for detecting unresolved (inner) disks and measuring the polarization of substellar companions. We have incorporated the correction method in a highly-automatic end-to-end data-reduction pipeline called IRDAP which is publicly available at https://irdap.readthedocs.io.