Recently, large samples of visually classified early-type galaxies (ETGs) containing dust have been identified using space-based infrared observations with the Herschel Space Telescope. The presence of large quantities of dust in massive ETGs is peculiar as X-ray halos of these galaxies are expected to destroy dust in 10 Myr (or less). This has sparked a debate regarding the origin of the dust: is it internally produced by asymptotic giant branch (AGB) stars, or is it accreted externally through mergers? We examine the 2D stellar and ionised gas kinematics of dusty ETGs using IFS observations from the SAMI galaxy survey, and integrated star-formation rates, stellar masses, and dust masses from the GAMA survey. Only 8% (4/49) of visually-classified ETGs are kinematically consistent with being dispersion-supported systems. These dispersion-dominated galaxies exhibit discrepancies between stellar and ionised gas kinematics, either offsets in the kinematic position angle or large differences in the rotational velocity, and are outliers in star-formation rate at a fixed dust mass compared to normal star-forming galaxies. These properties are suggestive of recent merger activity. The remaining 90% of dusty ETGs have low velocity dispersions and/or large circular velocities, typical of rotation-dominated galaxies. These results, along with the general evidence of published works on X-ray emission in ETGs, suggest that they are unlikely to host hot, X-ray gas consistent with their low stellar mass when compared to dispersion-dominated galaxies. This means dust will be long lived and thus these galaxies do not require external scenarios for the origin of their dust content.