We have investigated the electronic structure of BiS$_2$-based CeO$_{0.5}$F$_{0.5}$BiS$_2$ superconductor using polarization-dependent angle-resolved photoemission spectroscopy (ARPES), and succeeded in elucidating the orbital characters on the Fermi surfaces. In the rectangular Fermi pockets around X point, the straight portion parallel to the $k_y$ direction is dominated by Bi $6p_x$ character. The orbital polarization indicates the underlying quasi-one-dimensional electronic structure of the BiS$_2$ system. Moreover, distortions on tetragonally aligned Bi could give rise to the band Jahn-Teller effect.