We obtain Keck HIRES spectroscopy of HVS5, one of the fastest unbound stars in the Milky Way halo. We show that HVS5 is a 3.62 +- 0.11 Msun main sequence B star at a distance of 50 +- 5 kpc. The difference between its age and its flight time from the Galactic center is 105 +-18(stat)+-30(sys) Myr; flight times from locations elsewhere in the Galactic disk are similar. This 10^8 yr `arrival time between formation and ejection is difficult to reconcile with any ejection scenario involving massive stars that live for only 10^7 yr. For comparison, we derive arrival times of 10^7 yr for two unbound runaway B stars, consistent with their disk origin where ejection results from a supernova in a binary system or dynamical interactions between massive stars in a dense star cluster. For HVS5, ejection during the first 10^7 yr of its lifetime is ruled out at the 3-sigma level. Together with the 10^8 yr arrival times inferred for three other well-studied hypervelocity stars (HVSs), these results are consistent with a Galactic center origin for the HVSs. If the HVSs were indeed ejected by the central black hole, then the Galactic center was forming stars ~200 Myr ago, and the progenitors of the HVSs took ~100 Myr to enter the black holes loss cone.