In this work we present the effective field theory of primordial statistical anisotropies generated during anisotropic inflation involving a background $U(1)$ gauge field. Besides the usual Goldstone boson associated with the breaking of time diffeomorphism we have two additional Goldstone bosons associated with the breaking of spatial diffeomorphisms. We further identify these two new Goldstone bosons with the expected two transverse degrees of the $U(1)$ gauge field fluctuations. Upon defining the appropriate unitary gauge, we present the most general quadratic action which respects the remnant symmetry in the unitary gauge. The interactions between various Goldstone bosons leads to statistical anisotropy in curvature perturbation power spectrum. Calculating the general results for power spectrum anisotropy, we recover the previously known results in specific models of anisotropic inflation. In addition, we present novel results for statistical anisotropy in models with non-trivial sound speed for inflaton fluctuations. Also we identify the interaction which leads to birefringence-like effects in anisotropic power spectrum in which the speed of gauge field fluctuations depends on the direction of the mode propagation and the two polarization of gauge field fluctuations contribute differently in statistical anisotropy. As another interesting application, our EFT approach naturally captures interactions generating parity violating statistical anisotropies.