Group algebras of permutations have proved highly useful in solving a number of problems in large N gauge theories. I review the use of permutations in classifying gauge invariants in one-matrix and multi-matrix models and computing their correlators. These methods are also applicable to tensor models and have revealed a link between tensor models and the counting of branched covers. The key idea is to parametrize $U(N)$ gauge invariants using permutations, subject to equivalences. Correlators are related to group theoretic properties of these equivalence classes. Fourier transformation on symmetric groups by means of representation theory offers nice bases of functions on these equivalence classes. This has applications in AdS/CFT in identifying CFT duals of giant gravitons and their perturbations. It has also lead to general results on quiver gauge theory correlators, uncovering links to two dimensional topological field theory and the combinatorics of trace monoids.