The definitions of the $n^{th}$ Gauss sum and the associated $n^{th}$ central charge are introduced for premodular categories $mathcal{C}$ and $ninmathbb{Z}$. We first derive an expression of the $n^{th}$ Gauss sum of a modular category $mathcal{C}$, for any integer $n$ coprime to the order of the T-matrix of $mathcal{C}$, in terms of the first Gauss sum, the global dimension, the twist and their Galois conjugates. As a consequence, we show for these $n$, the higher Gauss sums are $d$-numbers and the associated central charges are roots of unity. In particular, if $mathcal{C}$ is the Drinfeld center of a spherical fusion category, then these higher central charges are 1. We obtain another expression of higher Gauss sums for de-equivariantization and local module constructions of appropriate premodular and modular categories. These expressions are then applied to prove the Witt invariance of higher central charges for pseudounitary modular categories.