We investigate the electronic structure of CaFe$_2$As$_2$ using high resolution photoemission spectroscopy. Experimental results exhibit three energy bands crossing the Fermi level making hole pockets around the $Gamma$-point. Temperature variation reveal a gradual shift of an energy band away from the Fermi level with the decrease in temperature in addition to the spin density wave (SDW) transition induced Fermi surface reconstruction of the second energy band across SDW transition temperature. The hole pocket in the former case eventually disappears at lower temperatures while the hole Fermi surface of the third energy band possessing finite $p$ orbital character survives till the lowest temperature studied. These results reveal signature of a complex charge redistribution among various energy bands as a function of temperature that might be associated to the exotic properties of this system.