Erbium ions doped into crystals have unique properties for quantum information processing, because of their optical transition at 1.5 $mu$m and of the large magnetic moment of their effective spin-1/2 electronic ground state. Most applications of erbium require however long electron spin coherence times, and this has so far been missing. Here, by selecting a host matrix with a low nuclear-spin density (CaWO$_4$) and by quenching the spectral diffusion due to residual paramagnetic impurities at millikelvin temperatures, we obtain an Er$^{3+}$ electron spin coherence time of 23 ms. This is the longest electron spin coherence time measured in a material with a natural abundance of nuclear spins and on a magnetically-sensitive transition. Our results establish Er$^{3+}$:CaWO$_4$ as a leading platform for quantum networks.