A standard method to study the mass distribution in galaxy clusters is through strong lensing of background galaxies in which the positions of multiple images of the same source constrain the surface mass distribution of the cluster. However, current parametrized mass models can often only reproduce the observed positions to within one or a few arcsec which is worse than the positional measurement uncertainty. One suggested explanation for this discrepancy is the additional perturbations of the path of the light ray caused by matter density fluctuations along the line of sight. We investigate this by calculating the statistical expectation value for the angular deflections caused by density fluctuations, which can be done given the matter power spectrum. We find that density fluctuations can, indeed, produce deflections of a few arcsec. We also find that the deflection angle of a particular image is expected to increase with source redshift and with the angular distance on the sky to the lens. Since the light rays of neighbouring images pass through much the same density fluctuations, it turns out that the images expected deflection angles can be highly correlated. This implies that line-of-sight density fluctuations are a significant and possibly dominant systematic for strong lensing mass modeling and set a lower limit to how well a cluster mass model can be expected to replicate the observed image positions. We discuss how the deflections and correlations should explicitly be taken into account in the mass model fitting procedure.