In a phase with fractional excitations, topological properties are enriched in the presence of global symmetry. In particular, fractional excitations can transform under symmetry in a fractionalized manner, resulting in different Symmetry Enriched Topological (SET) phases. While a good deal is now understood in $2D$ regarding what symmetry fractionalization patterns are possible, the situation in $3D$ is much more open. A new feature in $3D$ is the existence of loop excitations, so to study $3D$ SET phases, first we need to understand how to properly describe the fractionalized action of symmetry on loops. Using a dimensional reduction procedure, we show that these loop excitations exist as the boundary between two $2D$ SET phases, and the symmetry action is characterized by the corresponding difference in SET orders. Moreover, similar to the $2D$ case, we find that some seemingly possible symmetry fractionalization patterns are actually anomalous and cannot be realized strictly in $3D$. We detect such anomalies using the flux fusion method we introduced previously in $2D$. To illustrate these ideas, we use the $3D$ $Z_2$ gauge theory with $Z_2$ global symmetry as an example, and enumerate and describe the corresponding SET phases. In particular, we find four non-anomalous SET phases and one anomalous SET phase, which we show can be realized as the surface of a $4D$ system with symmetry protected topological order.