$beta$-Ga$_2$O$_3$ is a promising ultra-wide bandgap semiconductor whose properties can be further enhanced by alloying with Al. Here, using atomic-resolution scanning transmission electron microscopy (STEM), we find the thermodynamically-unstable $gamma$-phase is a ubiquitous defect in both $beta$-(Al$_x$Ga$_{1text{-}x}$)$_2$O$_3$ films and doped $beta$-Ga$_2$O$_3$ films grown by molecular beam epitaxy. For undoped $beta$-(Al$_x$Ga$_{1text{-}x}$)$_2$O$_3$ films we observe $gamma$-phase inclusions between nucleating islands of the $beta$-phase at lower growth temperatures (~400-600 $^{circ}$C). In doped $beta$-Ga$_2$O$_3$, a thin layer of the $gamma$-phase is observed on the surfaces of films grown with a wide range of n-type dopants and dopant concentrations. The thickness of the $gamma$-phase layer was most strongly correlated with the growth temperature, peaking at about 600 $^{circ}$C. Ga interstitials are observed in $beta$-phase, especially near the interface with the $gamma$-phase. By imaging the same region of the surface of a Sn-doped $beta$-(Al$_x$Ga$_{1text{-}x}$)$_2$O$_3$ after ex-situ heating up to 400 $^{circ}$C, a $gamma$-phase region is observed to grow above the initial surface, accompanied by a decrease in Ga interstitials in the $beta$-phase. This suggests that the diffusion of Ga interstitials towards the surface is likely the mechanism for growth of the surface $gamma$-phase, and more generally that the more-open $gamma$-phase may offer diffusion pathways to be a kinetically-favored and early-forming phase in the growth of Ga$_2$O$_3$.