We compare the most successful and widely used map of Galactic dust extinction, provided by Schlegel, Finkbeiner & Davis (1998; hereafter SFD), to the galaxy number counts in the Sloan Digital Sky Survey (SDSS) photometric/spectroscopic DR4 sample. We divide the SDSS survey area into 69 disjoint subregions according to the dust extinction provided by SFD and compare the surface number density of galaxies in each subregion. As expected, the galaxy surface number density decreases with increasing extinction but only for SFD extinction values above about 0.1 to 0.2 magnitudes (depending on the band). At lower values of the SFD extinction, we find that the sky surface density of galaxies increases with increasing extinction, precisely the opposite of the effect expected from Galactic dust. We suggest that the far infrared (FIR) brightness of the sky in regions of true low dust extinction is significantly ``contaminated by the FIR emission from background galaxies. We show that such an explanation is both qualitatively and quantitatively consistent with the available data. Based on this interpretation we conclude that systematic errors in the SFD extinction map due to extragalactic FIR emission are quite small, of order hundredths of a magnitude, but nevertheless statistically detectable. (Abridged)