We report on the results of an XMM-Newton observation of the Supergiant Fast X-ray Transient (SFXT) IGR J08408-4503 performed in June 2020. The source is composed by a compact object (likely a neutron star) orbiting around an O8.5Ib-II(f)p star, LM Vel. The X-ray light curve shows a very low level of emission, punctuated by a single, faint flare. Analysis of spectra measured during the flare and during quiescence is performed. The quiescent state shows a continuum spectrum well deconvolved to three spectral models: two components are from a collisionally-ionized plasma (with temperatures kT1=0.24 keV and kT2=0.76 keV), together with a power law model (photon index of 2.55), dominating above 2 keV. The X-ray flux emitted at this lowest level is 3.2$times10^{-13}$ erg/cm2/s (0.5-10 keV, corrected for the interstellar absorption), implying an X-ray luminosity of 1.85$times10^{32}$ erg/s (at 2.2 kpc). The two temperature collisionally-ionized plasma is intrinsic to the stellar wind of the donor star, while the power law can be interpreted as emission due to residual, low level accretion onto the compact object. The X-ray luminosity contributed by the power law component only, in the lowest state, is (4.8$pm{1.4})times10^{31}$ erg/s, the lowest quiescent luminosity detected from the compact object in an SFXT. Thanks to this very faint X-ray state caught by XMM-Newton, X-ray emission from the wind of the donor star LM Vel could be well-established and studied in detail for the first time, as well as a very low level of accretion onto the compact object. The residual accretion rate onto the compact object in IGR J08408-4503 can be interpreted as the Bohm diffusion of (possibly magnetized) plasma entering the neutron star magnetosphere at low Bondi capture rates from the supergiant donor wind at the quasi-spherical radiation-driven settling accretion stage.