The layered topological insulator MnBi$_2$Te$_4$ has attracted great interest recently due to its intrinsic antiferromagnetic order, potentially hosting various topological phases. By temperature-dependent infrared spectroscopy over a broad frequency range, we studied the changes in the optical conductivity of MnBi$_2$Te$_4$ at the magnetic ordering temperature. The temperature dependence of several optical parameters reveals an anomaly at the magnetic phase transition, which suggests the correlation between the bulk electronic band structure and the magnetism. We relate our findings to recent reports on the temperature dependence of the electronic band structure of MnBi$_2$Te$_4$.