The B modes generated by the lensing of CMB polarization are a primary target for the upcoming generation of experiments and can potentially constrain quantities such as the neutrino mass and dark energy equation of state. The net sample variance on the small scale B modes out to l=2000 exceeds Gaussian expectations by a factor of 10 reflecting the variance of the larger scale lenses that generate them. It manifests itself as highly correlated band powers with correlation coefficients approaching 70% for wide bands of Delta l/l ~0.25. It will double the total variance for experiments that achieve a sensitivity of approximately 4 uK-arcmin and a beam of several arcminutes or better. This non-Gaussianity must be taken into account in the analysis of experiments that go beyond first detection.