Lorentz invariance plays a fundamental role in modern physics. However, tiny violations of the Lorentz invariance may arise in some candidate quantum gravity theories. Prominent signatures of the gravitational Lorentz invariance violation (gLIV) include anisotropy, dispersion, and birefringence in the dispersion relation of gravitational waves (GWs). Using a total of 50 GW events in the GW transient catalogs GWTC-1 and GWTC-2, we perform an analysis on the anisotropic birefringence phenomenon. The use of multiple events allows us to completely break the degeneracy among gLIV coefficients and globally constrain the coefficient space. Compared to previous results at mass dimensions 5 and 6 for the Lorentz-violating operators, we tighten the global limits of 34 coefficients by factors ranging from $2$ to $7$.