Hexagonal YMnO$_3$ is well known for the co-occurrence of ferroelectricity and antiferromagnetism at low temperatures. Using temperature-dependent spectroscopic ellipsometry at an $a$-plane oriented single crystal, we show how the dielectric function
is affected by the magnetic order transition at the Neel temperature. We focus especially on the pronounced charge transfer transitions around (1.6-1.7)eV which are strongly connected to Mn 3$d$ electrons. If described with a Bose-Einstein model, the temperature dependency of their energy and broadening is characterized by effective phonon energies not larger than 8meV. We argue that this is a hint for the occurrence of a soft phonon mode related to the antiferromagnetic phase transition. This is observed in both tensor components of the dielectric function, parallel and perpendicular to the crystallographic $c$-axis. Furthermore, a suitable parametrization for the uniaxial dielectric function is presented for the NIR-VUV spectral range. The broad transitions at energies higher than a critical point-like bandgap do not show a clear temperature dependence. We also observe some weak discrete absorption features around the strong charge transfer transitions with energies matching well to low-temperature photoluminescence signals.
