We have investigated the electronic structure of polycrystalline Ca$_2$FeReO$_6$ using photoemission spectroscopy and band-structure calculations within the local-density approximation+$U$ (LDA+$U$) scheme. In valence-band photoemission spectra, a double-peak structure which is characteristic of the metallic double perovskite series has been observed near the Fermi level ($E_{rm F}$), although it is less distinct compared to the Sr$_2$FeMoO$_6$ case. The leading near-$E_{rm F}$ structure has a very weak spectral weight at $E_{rm F}$ above the metal-insulator transition (MIT) temperature $T_{rm MI}$ of $sim$140 K, and it loses the $E_{rm F}$ weight below $T_{rm MI}$, forming a small energy gap. To reproduce this small energy gap in the calculation, we require a very large effective $U$ ($U_{rm eff}$) for Re (4 eV) in addition to a relatively large $U_{rm eff}$ for Fe (4 eV). Although the most of the experimental features can be interpreted with the help of the band theory, the overall agreement between the theory and the experiment was not satisfactory. We demonstrate that the effective transfer integral between Fe and Re is actually smaller than that between Fe and Mo in Ca$_2$FeMoO$_6$, which can explain both MIT and very high ferrimagnetic transition temperature.