We propose the $mathbb{Z}_Q$ Berry phase as a topological invariant for higher-order symmetry-protected topological (HOSPT) phases for two- and three-dimensional systems. It is topologically stable for electron-electron interactions assuming the gap remains open. As a concrete example, we show that the Berry phase is quantized in $mathbb{Z}_4$ and characterizes the HOSPT phase of the extended Benalcazar-Bernevig-Hughes (BBH) model, which contains the next-nearest neighbor hopping and the intersite Coulomb interactions. In addition, we introduce the $mathbb{Z}_4$ Berry phase for the spin-model-analog of the BBH model, whose topological invariant has not been found so far. Furthermore, we demonstrate the Berry phase is quantized in $mathbb{Z}_4$ for the three-dimensional version of the BBH model. We also confirm the bulk-corner correspondence between the $mathbb{Z}_4$ Berry phase and the corner states in the HOSPT phases.