We investigate the environment and line of sight of the H0LiCOW lens B1608+656 using Subaru Suprime-Cam and the Hubble Space Telescope (HST) to perform a weak lensing analysis. We compare three different methods to reconstruct the mass map of the field, i.e. the standard Kaiser-Squires inversion coupled with inpainting and Gaussian or wavelet filtering, and $tt{Glimpse}$ a method based on sparse regularization of the shear field. We find no substantial difference between the 2D mass reconstructions, but we find that the ground-based data is less sensitive to small-scale structures than the space-based observations. Marginalising over the results obtained with all the reconstruction techniques applied to the two available HST filters F606W and F814W, we estimate the external convergence, $kappa_{rm ext}$ at the position of B1608+656 is $kappa_{rm ext} = 0.11^{+0.06}_{-0.04}$, where the error bars corresponds respectively to the 16th and 84th quartiles. This result is compatible with previous estimates using the number-counts technique, suggesting that B1608+656 resides in an over-dense line of sight, but with a completely different technique. Using our mass reconstructions, we also compare the convergence at the position of several groups of galaxies in the field of B1608+656 with the mass measurements using various analytical mass profiles, and find that the weak lensing results favor truncated halo models.