Accurate weak lensing mass estimates of clusters are needed in order to calibrate mass proxies for the cosmological exploitation of galaxy cluster surveys. Such measurements require accurate knowledge of the redshift distribution of the weak lensing source galaxies. In this context, we investigate the accuracy of photometric redshifts (photo-$z$s) computed by the 3D-HST team for the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey fields, which provide a relevant photometric reference data set for deep weak lensing studies. Through the comparison to spectroscopic redshifts and photo-$z$s based on very deep data from the Hubble Ultra Deep Field, we identify catastrophic redshift outliers in the 3D-HST/CANDELS catalogue. These would significantly bias weak lensing results if not accounted for. We investigate the cause of these outliers and demonstrate that the interpolation of spectral energy distribution (SED) templates and a well-selected combination of photometric data can reduce the net impact for weak lensing studies.