J-PET is the first positron-emission tomograph (PET) constructed from plastic scintillators. It was optimized for the detection of photons from electron-positron annihilation. Such photons, having an energy of 511 keV, interact with electrons in plastic scintillators predominantly via the Compton effect. Compton scattering is at most probable at an angle orthogonal to the electric field vector of the interacting photon. Thus registration of multiple photon scatterings with J-PET enables to determine the polarization of the annihilation photons. In this contribution we present estimates on the physical limitation in the accuracy of the polarization determination of $511$~keV photons with the J-PET detector.