The Fermi bubbles were possibly created by large injections of energy into the Galactic Center (GC), either by an active galactic nucleus (AGN) or by nuclear starburst more than ~10 Myr ago. However, the origin of the diffuse gamma-ray emission associated with Loop I, a radio continuum loop spanning across 100 deg on the sky, is still being debated. The northern-most part of Loop I, known as the North Polar Spur (NPS), is the brightest arm and is even clearly visible in the ROSAT X-ray sky map. In this paper, we present a comprehensive review on the X-ray observations of the Fermi bubbles and their possible association with the NPS and Loop I structures. Using uniform analysis of archival Suzaku and Swift data, we show that X-ray plasma with kT ~ 0.3 keV and low metal abundance (Z ~ 0.2 Z_solar) is ubiquitous in both the bubbles and Loop I and is naturally interpreted as weakly shock-heated Galactic halo gas. However, the observed asymmetry of the X-ray-emitting gas above and below the GC has still not been resolved; it cannot be fully explained by the inclination of the axis of the Fermi bubbles to the Galactic disk normal. We argue that the NPS and Loop I may be asymmetric remnants of a large explosion that occurred before the event that created the Fermi bubbles, and that the soft gamma-ray emission from Loop I may be due to either pi^0 decay of accelerated protons or electron bremsstrahlung.