The giant planets of our solar system possess envelopes consisting mainly of hydrogen and helium but are also significantly enriched in heavier elements relatively to our Sun. In order to better constrain how these heavy elements have been delivered, we quantify the amount accreted during the so-called late heavy bombardment, at a time when planets were fully formed and planetesimals could not sink deep into the planets. On the basis of the Nice model, we obtain accreted masses (in terrestrial units) equal to $0.15pm0.04 rm,M_oplus$ for Jupiter, and $0.08 pm 0.01 rm,M_oplus$ for Saturn. For the two other giant planets, the results are found to depend mostly on whether they switched position during the instability phase. For Uranus, the accreted mass is $0.051 pm 0.003 rm,M_oplus$ with an inversion and $0.030 pm 0.001 rm,M_oplus$ without an inversion. Neptune accretes $0.048 pm 0.015 rm,M_oplus$ in models in which it is initially closer to the Sun than Uranus, and $0.066 pm 0.006 rm,M_oplus$ otherwise. With well-mixed envelopes, this corresponds to an increase in the enrichment over the solar value of $0.033 pm 0.001$ and $0.074 pm 0.007$ for Jupiter and Saturn, respectively. For the two other planets, we find the enrichments to be $2.1 pm 1.4$ (w/ inversion) or $1.2 pm 0.7$ (w/o inversion) for Uranus, and $2.0 pm 1.2$ (w/ inversion) or $2.7 pm 1.6$ (w/o inversion) for Neptune. This is clearly insufficient to explain the inferred enrichments of $sim 4$ for Jupiter, $sim 7$ for Saturn and $sim 45$ for Uranus and Neptune.