During the inspiral and merger of a binary black hole, gravitational radiation is emitted anisotropically due to asymmetries in the merger configuration. This anisotropic radiation leads to a gravitational wave kick, or recoil velocity, as large as ~ 4000 km/sec. We investigate the effect gravitational recoil has on the retention of intermediate mass black holes (IMBH) within Galactic globular clusters. Assuming that our current understanding of IMBH-formation is correct and yields an IMBH-seed in every globular cluster, we find a significant problem retaining low mass IMBHs (1000 $Msun$) in the typical merger-rich globular cluster environment. Given a uniform black hole spin distribution and orientation and a Kroupa IMF, we find that at most 3% of the globular clusters can retain an IMBH larger than 1000 $Msun$ today. For a population of black holes that better approximates mass loss from winds and supernovae, we find that 16% of globulars can retain an IMBH larger than 1000 $Msun$. Our calculations show that if there are black holes of mass $M > 60 Msun$ in a cluster, repeated IMBH-BH encounters will eventually eject a 1000 $Msun$ IMBH with greater than 30% probability. As a consequence, a large population of rogue black holes may exist in our Milky Way halo. We discuss the dynamical implications of this subpopulation, and its possible connection to ultraluminous X-ray sources (ULXs).