The modern theory of electric polarization has recently been extended to higher multipole moments, such as quadrupole and octupole moments. The higher electric multipole insulators are essentially topological crystalline phases protected by underlying crystalline symmetries. Henceforth, it is natural to ask what are the consequences of symmetry breaking in these higher multipole insulators. In this work, we investigate topological phases and the consequences of symmetry breaking in generalized electric quadrupole insulators. Explicitly, we generalize the Benalcazar-Bernevig-Hughes model by adding specific terms in order to break the crystalline and non-spatial symmetries. Our results show that chiral symmetry breaking induces an indirect gap phase which hides corner modes in bulk bands, ruining the topological quadrupole phase. We also demonstrate that quadrupole moments can remain quantized even when mirror symmetries are absent in a generalized model. Furthermore, it is shown that topological quadrupole phase is robust against a unique type of disorder presented in the system.