A braided monoidal category may be considered a $3$-category with one object and one $1$-morphism. In this paper, we show that, more generally, $3$-categories with one object and $1$-morphisms given by elements of a group $G$ correspond to $G$-crossed braided categories, certain mathematical structures which have emerged as important invariants of low-dimensional quantum field theories. More precisely, we show that the 4-category of $3$-categories $mathcal{C}$ equipped with a 3-functor $mathrm{B}G to mathcal{C}$ which is essentially surjective on objects and $1$-morphisms is equivalent to the $2$-category of $G$-crossed braided categories. This provides a uniform approach to various constructions of $G$-crossed braided categories.