For the interpretation of measurements of radio emission from extensive air showers, an important systematic uncertainty arises from natural variations of the atmospheric refractive index $n$. At a given altitude, the refractivity $N=10^6, (n-1)$ can have relative variations on the order of $10 %$ depending on temperature, humidity, and air pressure. Typical corrections to be applied to $N$ are about $4%$. Using CoREAS simulations of radio emission from air showers, we have evaluated the effect of varying $N$ on measurements of the depth of shower maximum $X_{rm max}$. For an observation band of 30 to 80 MHz, a difference of $4 %$ in refractivity gives rise to a systematic error in the inferred $X_{rm max}$ between 3.5 and 11 $mathrm{g/cm^2}$, for proton showers with zenith angles ranging from 15 to 50 degrees. At higher frequencies, from 120 to 250 MHz, the offset ranges from 10 to 22 $mathrm{g/cm^2}$. These offsets were found to be proportional to the geometric distance to $X_{rm max}$. We have compared the results to a simple model based on the Cherenkov angle. For the 120 to 250 MHz band, the model is in qualitative agreement with the simulations. In typical circumstances, we find a slight decrease in $X_{rm max}$ compared to the default refractivity treatment in CoREAS. While this is within commonly treated systematic uncertainties, accounting for it explicitly improves the accuracy of $X_{rm max}$ measurements.