We dispersively interface an ensemble of one thousand atoms trapped in the evanescent field surrounding a tapered optical nanofiber. This method relies on the azimuthally-asymmetric coupling of the ensemble with the evanescent field of an off-resonant probe beam, transmitted through the nanofiber. The resulting birefringence and dispersion are significant; we observe a phase shift per atom of $sim$,1,mrad at a detuning of six times the natural linewidth, corresponding to an effective resonant optical density per atom of 0.027. Moreover, we utilize this strong dispersion to non-destructively determine the number of atoms.