We investigated the evolution of ferromagnetism in layered Fe$_3$GeTe$_2$ flakes under different pressures and temperatures using in situ magnetic circular dichroism (MCD) spectroscopy. We found that the rectangle shape of hysteretic loop under an out-of-plane magnetic field sweep can sustain below 7 GPa. Above that pressure, an intermediate state appears at low temperature region signaled by an 8-shaped skew hysteretic loop. Meanwhile, the coercive field and Curie temperature decrease with increasing pressures, implying the decrease of the exchange interaction and the magneto-crystalline anisotropy under pressures. The intermediate phase has a labyrinthine domain structure, which is attributed to the increase of ratio of exchange interaction to magneto-crystalline anisotropy based on Jaglas theory. Moreover, our calculation results reveal a weak structural transition around 6 GPa, which leads to a drop of the magnetic momentum of Fe ions.