We present the first detailed kinematic analysis of the proper motions (PMs) of stars in the Magellanic Bridge, from both the textit{Gaia} Data Release 2 catalog and from textit{Hubble Space Telescope} Advanced Camera for Surveys data. For the textit{Gaia} data, we identify and select two populations of stars in the Bridge region, young main sequence (MS) and red giant stars. The spatial locations of the stars are compared against the known H {small I} gas structure, finding a correlation between the MS stars and the H {small I} gas. In the textit{Hubble Space Telescope} fields our signal comes mainly from an older MS and turn-off population, and the proper motion baselines range between $sim 4$ and 13 years. The PMs of these different populations are found to be consistent with each other, as well as across the two telescopes. When the absolute motion of the Small Magellanic Cloud is subtracted out, the residual Bridge motions display a general pattern of pointing away from the Small Magellanic Cloud towards the Large Magellanic Cloud. We compare in detail the kinematics of the stellar samples against numerical simulations of the interactions between the Small and Large Magellanic Clouds, and find general agreement between the kinematics of the observed populations and a simulation in which the Clouds have undergone a recent direct collision.