To resolve the nature of the hidden order below 17.5,K in the heavy fermion compound URu$_2$Si$_2$, identifying which symmetries are broken below the hidden order transition is one of the most important steps. Several recent experiments on the electronic structure have shown that the Fermi surface in the hidden order phase is quite close to the result of band-structure calculations within the framework of itinerant electron picture assuming the antiferromagnetism. This provides strong evidence for the band folding along the c-axis with the ordering vector of $(0,0,1)$, corresponding to broken translational symmetry. In addition to this, there is growing evidence for fourfold rotational symmetry breaking in the hidden-order phase from measurements of the in-plane magnetic anisotropy and the effective mass anisotropy in the electronic structure, as well as the orthorhombic lattice distortion. This broken fourfold symmetry gives a stringent constraint that the symmetry of the hidden order parameter should belong to the degenerate $E$-type irreducible representation. We also discuss a possibility that time reversal symmetry is also broken, which further narrows down the order parameter that characterizes the hidden order.