A stochastic gravitational wave background (SGWB) would gravitationally lens the cosmic microwave background (CMB) photons. We find that the lensing due to gravitational waves(GW) is more efficient as compared to lensing due to scalar density perturbations. Though the effect of lensing due to GW is found to effect all the four CMB power spectra, its effect is most prominently seen in the CMB polarization power spectra. This suggests that the measurements of the CMB angular power spectra could be used to constraining the energy density ($Omega_{GW}$) of the SGBW. In our analysis we find that the most stringent constraints on $Omega_{GW}$ are due to measurements of the angular power spectra of CMB temperature anisotropies. We show that in the future it will be possible to place more stringent bounds on $Omega_{GW}$ using improved upper limits or detections of the angular power spectra of the B-modes of CMB polarization at large multipoles.