We report an optical homogeneous linewidth of 580 $pm$ 20 Hz of Er$^{3+}$:Y$_2$O$_3$ ceramics at millikelvin temperatures, narrowest so far in rare-earth doped ceramics. We show slow spectral diffusion of $sim$2 kHz over a millisecond time scale. Temperature, field dependence of optical coherence and spectral diffusions reveal the remaining dephasing mechanism as elastic two-level systems in polycrystalline grain boundaries and superhyperfine interactions of Er$^{3+}$ with nuclear spins. In addition, we perform spectral holeburning and measure up to 5 s hole lifetimes. These spectroscopic results put Er$^{3+}$:Y$_2$O$_3$ ceramics as a promising candidate for telecommunication quantum memories and light-matter interfaces.