The electronic structure of Cr$_{1/3}$NbSe$_2$ is studied via optical spectroscopy. We observe two low-energy interband transitions in the paramagnetic phase, which split into four peaks as the compound enters the ferromagnetic state. The band structure calculation indicates the four peaks are interband transitions to the spin up Cr e$_g$ states. We show that the peak splitting below the Curie temperature is emph{not} due to the exchange splitting of spin up and down bands, but directly reflects a band broadening effect in Cr-derived states upon the spontaneous ferromagnetic ordering.