The model of holographic dark energy (HDE) with massive neutrinos and/or dark radiation is investigated in detail. The background and perturbation evolutions in the HDE model are calculated. We employ the PPF approach to overcome the gravity instability difficulty (perturbation divergence of dark energy) led by the equation-of-state parameter $w$ evolving across the phantom divide $w=-1$ in the HDE model with $c<1$. We thus derive the evolutions of density perturbations of various components and metric fluctuations in the HDE model. The impacts of massive neutrino and dark radiation on the CMB anisotropy power spectrum and the matter power spectrum in the HDE scenario are discussed. Furthermore, we constrain the models of HDE with massive neutrinos and/or dark radiation by using the latest measurements of expansion history and growth of structure, including the Planck CMB temperature data, the baryon acoustic oscillation data, the JLA supernova data, the Hubble constant direct measurement, the cosmic shear data of weak lensing, the Planck CMB lensing data, and the redshift space distortions data. We find that $sum m_ u<0.186$ eV (95% CL) and $N_{rm eff}=3.75^{+0.28}_{-0.32}$ in the HDE model from the constraints of these data.