We construct a state-sum type invariant of smooth closed oriented $4$-manifolds out of a $G$-crossed braided spherical fusion category ($G$-BSFC) for $G$ a finite group. The construction can be extended to obtain a $(3+1)$-dimensional topological quantum field theory (TQFT). The invariant of $4$-manifolds generalizes several known invariants in literature such as the Crane-Yetter invariant from a ribbon fusion category and Yetters invariant from homotopy $2$-types. If the $G$-BSFC is concentrated only at the sector indexed by the trivial group element, a cohomology class in $H^4(G,U(1))$ can be introduced to produce a different invariant, which reduces to the twisted Dijkgraaf-Witten theory in a special case. Although not proven, it is believed that our invariants are strictly different from other known invariants. It remains to be seen if the invariants are sensitive to smooth structures. It is expected that the most general input to the state-sum type construction of $(3+1)$-TQFTs is a spherical fusion $2$-category. We show that a $G$-BSFC corresponds to a monoidal $2$-category with certain extra structure, but that structure does not satisfy all the axioms of a spherical fusion $2$-category given by M. Mackaay. Thus the question of what axioms properly define a spherical fusion $2$-category is open.