We describe a weak lensing view of the downsizing of star forming galaxies based on cross correlating a weak lensing ($kappa$) map with a predicted map constructed from a redshift survey. Moderately deep and high resolution images with Subaru/Hyper Suprime-Cam covering the 4 deg^2 DLS F2 field provide a $kappa$ map with 1 arcmin resolution. A dense complete redshift survey of the F2 field including 12,705 galaxies with $Rleq20.6$ is the basis for construction of the predicted map. The zero-lag cross-correlation between the kappa and predicted maps is significant at the $30sigma$ level. The width of the cross-correlation peak is comparable with the angular scale of rich cluster at $zsim0.3$, the median depth of the redshift survey. Slices of the predicted map in $delta{z} = 0.05$ redshift bins enable exploration of the impact of structure as a function of redshift. The zero-lag normalised cross-correlation has significant local maxima at redshifts coinciding with known massive X-ray clusters. Even in slices where there are no known massive clusters, there is significant signal in the cross-correlation originating from lower mass groups that trace the large-scale of the universe. Spectroscopic $D_n4000$ measurements enable division of the sample into star-forming and quiescent populations. The significance of the cross-correlation with structure containing star-forming galaxies increases with redshift from $5sigma$ at $z = 0.3$ to $7 sigma$ at $z = 0.5$. The weak lensing results are consistent with the downsizing view of galaxy evolution established on the basis of many other independent studies.