We provide constraints on the AGN contribution to the mid-IR extragalactic background light from a correlation analysis of deep X-ray and mid-IR observations in two regions centred on the Lockman Hole (LH) and Hubble Deep Field North (HDFN). Among the 76 galaxies detected by XMM in the LH area (200 square arcminutes), 24 show mid-IR emission, but the relative percentage of X-ray sources with mid-IR counterparts increases with the band energy: from 30% of the 0.5-2 keV sources up to 63% of the 5-10 keV sources. In contrast, only a small fraction of the mid-IR sources (around 10%) show X-ray emission. In the region centred on the HDFN (24 square arcminutes), 25% of the mid-IR sources are detected in the X-ray, while 30-40% of the X-ray sources show mid-IR emission. Under the conservative assumption that all XMM sources are AGN-dominated, AGNs contribute 15% of the total mid-IR flux in the LH. For the HDFN we have assumed that AGN-dominated sources are luminous X-ray sources with radio to X-ray SEDs typical of local AGNs, in which case we find that 18% of the mid-IR flux are due to AGN emission. If we put together all the existing information from the deepest HDFN data to the shallow large-area BeppoSAX observations in the ELAIS S1 region using the median mid-IR to X-ray spectral indices as a function of the X-ray flux, we find an AGN contribution to the 15um background of 17%. We conclude that the population of IR luminous galaxies detected in the ISOCAM deep surveys are mainly constituted by dust-obscured starbursts.