Palladium diselenide (PdSe$_2$), a new type of two-dimensional noble metal dihalides (NMDCs), has received widespread attention for its excellent electrical and optoelectronic properties. Herein, high-quality continuous centimeter-scale PdSe$_2$ films with layers in the range of 3L-15L were grown using Chemical Vapor Deposition (CVD) method. The absorption spectra and DFT calculations revealed that the bandgap of the PdSe$_2$ films decreased with increasing number of layers, which is due to PdSe$_2$ enhancement of orbital hybridization. Spectroscopic ellipsometry (SE) analysis shows that PdSe2 has significant layer-dependent optical and dielectric properties. This is mainly due to the unique strong exciton effect of the thin PdSe$_2$ film in the UV band. In particular, the effect of temperature on the optical properties of PdSe$_2$ films was also observed, and the thermo-optical coefficients of PdSe$_2$ films with different number of layers were calculated. This study provides fundamental guidance for the fabrication and optimization of PdSe$_2$-based optoelectronic devices.