Misalignment of gas and stellar rotation in galaxies can give clues to the origin and processing of accreted gas. Integral field spectroscopic observations of 1213 galaxies from the SAMI Galaxy Survey show that 11% of galaxies with fitted gas and stellar rotation are misaligned by more than 30 degrees in both field/group and cluster environments. Using SAMI morphological classifications and Sersic indices, the misalignment fraction is 45+/-6% in early-type galaxies, but only 5+/-1% in late-type galaxies. The distribution of position angle offsets is used to test the physical drivers of this difference. Slower dynamical settling time of the gas in elliptical stellar mass distributions accounts for a small increase in misalignment in early-type galaxies. However, gravitational dynamical settling time is insufficient to fully explain the observed differences between early- and late-type galaxies in the distributions of the gas/stellar position angle offsets. LTGs have primarily accreted gas close to aligned rather than settled from misaligned based on analysis of the skewed distribution of PA offsets compared to a dynamical settling model. Local environment density is less important in setting the misalignment fractions than morphology, suggesting that mergers are not the main source of accreted gas in these disks. Cluster environments are found to have gas misalignment driven primarily by cluster processes not by gas accretion.