We present X-ray observations of the transient accretion-powered millisecond pulsar IGR J00291+5934 during quiescence. IGR J00291+5934 is the first source among accretion powered millisecond pulsars to show signs of a thermal component in its quiescent spectrum. Fitting this component with a neutron star atmosphere or a black body model we obtain soft temperatures (~64 eV and ~110 eV, respectively). As in other sources of this class a hard spectral component is also present, comprising more than 60% of the unabsorbed 0.5-10 keV flux. Interpreting the soft component as cooling emission from the neutron star, we can conclude that the compact object can be spun up to milliscond periods by accreting only <0.2 solar masses.