We report the discovery of four groups of galaxies along the line of sight to the B1608+656 gravitational lens system. One group is at the redshift of the primary lensing galaxy (z = 0.631) and appears to have a low mass, with eight spectroscopically-confirmed members and an estimated velocity dispersion of 150+/-60 km/s. The three other groups are in the foreground of the lens. These groups contain ~10 confirmed members each, and are located at redshifts of 0.265, 0.426, and 0.520. Two of the three additional groups are centered roughly on the lens system, while the third is centered ~1 arcminute south of the lens. We find that each group provides an external convergence (kappa_ext) of ~0.005--0.060, depending on the assumptions made in calculating the convergence. For lens systems where no additional observables that can break the mass-sheet degeneracy exist, the determination of H_0 will be biased high by a factor of (1-kappa_ext)^{-1} if the external convergence is not properly included in the model. For the B1608+656 system, the stellar velocity dispersion of the lensing galaxy has been measured, thus breaking the mass-sheet degeneracy due to the group that is physically associated with the lens. The effect of the other groups along the line of sight can be folded into the overall uncertainties due to large-scale structure (LSS) along the line of sight. Because the B1608+656 system appears to lie along an overdense line of sight, the LSS will cause the measurement of H_0 to be biased high for this system. The systematic bias introduced by LSS could be 5% or greater. (Abridged)