We report on optical and NIR observations obtained during and after the 2004 December discovery outburst of the X-ray transient and accretion-powered millisecond pulsar IGR J00291+5934. Our observations monitored the evolution of the brightness and the spectral properties of J00291 during the outburst decay towards quiescence. We also present optical, NIR and Chandra observations obtained during true quiescence. Photometry of the field during outburst reveals an optical and NIR counterpart that brightened from R~23 to R~17 and from K=19 to K~16. Spectral analysis of the RIJHK broadband photometry shows excess in the NIR bands that may be due to synchrotron emission. The Halpha emission line profile suggests the orbital inclination is ~22-32 degrees. The preferred range for the reddening towards the source is 0.7 < E(B-V) < 0.9, which is equivalent to 4.06E21 cm^-2 < NH < 5.22E21 cm^-2. The Chandra observations of the pulsar in its quiescent state gave an unabsorbed 0.5-10 keV flux for the best-fitting power-law model to the source spectrum of (7.0 +/- 0.9)E-14 ergs/cm^2/s (adopting a hydrogen column of 4.6E21 cm^-2. The fit resulted in a power-law photon index of 2.4 +/- 0.5. The (R-K)o color observed during quiescence supports an irradiated donor star and accretion disk. We estimate a distance of 2 to 4 kpc towards J00291 by using the outburst X-ray light curve and the estimated critical X-ray luminosity necessary to keep the outer parts of the accretion disk ionized. Using the quiescent X-ray luminosity and the spin period, we constrain the magnetic field of the neutron star to be < 3E8 Gauss.