We report the anisotropic changes in the electronic structure of a Kondo semiconductor CeOs$_2$Al$_{10}$ across an anomalous antiferromagnetic ordering temperature ($T_0$) of 29 K, using optical conductivity spectra. The spectra along the $a$- and $c$-axes indicate that a $c$-$f$ hybridization gap emerges from a higher temperature continuously across $T_0$. Along the b-axis, on the other hand, a different energy gap with a peak at 20 meV appears below 39 K, which is higher temperature than $T_0$, because of structural distortion. The onset of the energy gap becomes visible below $T_0$. Our observation reveals that the electronic structure as well as the energy gap opening along the b-axis due to the structural distortion induces antiferromagnetic ordering below $T_0$.