We investigate the structure formation in the effective field theory of the holographic dark energy. The equation of motion for the energy contrast $delta_m$ of the cold dark matter is the same as the one in the general relativity up to the leading order in the small scale limit $kgg aH$, provided the equation of state is Quintessence-like. Our effective field theory breaks down while the equation of state becomes phantom-like. We propose a solution to this problem by eliminating the scalar graviton.