Near- and mid-IR survey data from DENIS and ISOGAL are used to investigate the structure and formation history of the inner 10 degree (1.4 kpc) of the Milky Way galaxy. Synthetic bolometric corrections and extinction coefficients in the near- and mid-IR are derived for stars of different spectral types, to allow the transformation of theoretical isochrones into observable colour-magnitude diagrams. The observed IR colour-magnitude diagrams are used to derive the extinction, metallicity and age for individual stars. The inner galaxy is dominated by an old population (> 7 Gyr). In addition, an intermediate-age population (200 Myr to 7 Gyr) is detected, which is consistent with the presence of a few hundred Asymptotic Giant Branch stars with heavy mass loss. Furthermore, young stars (< 200 Myr) are found across the inner Bulge. The metallicities of these stellar population components are discussed. These results can be interpreted in terms of an early epoch of intense star formation and chemical enrichment which shaped the bulk of the Bulge and nucleus, and a more continuous star formation history which gradually shaped the disk from the accretion of sub-solar metallicity gas from the halo. A possible increase in star formation about 200 Myr ago might have been triggered by a minor merger. Ever since the formation of the first stars, mechanisms have been at play that mix the populations from the nucleus, Bulge and disk. Luminosity functions across the inner galactic plane indicate the presence of an inclined (bar) structure at > 1 kpc from the galactic centre, near the inner Lindblad resonance. The innermost part of the Bulge, within about 1 kpc from the galactic centre, seems azimuthally symmetric.