We report the synthesis and superconducting properties of a new transition-metal chalcogenide Ta$_2$PdSe$_5$. The measurements of resistivity, magnetization, and specific heat reveal that Ta$_2$PdSe$_5$ is a bulk superconductor with $T_c$ $simeq$ 2.5 K. The zero-field electronic specific heat in the superconducting state can be fitted with a two-gap BCS model. The upper critical field $H_{c2}$ shows a linear temperature dependence, and the value of $H_{c2}$(0) is much higher than the estimated Pauli limiting field $H_{c2}^{P}$ and orbital limiting field $H_{c2}^{orb}$. All these results of specific heat and upper critical field suggest that Ta$_2$PdSe$_5$ is a multi-band superconductor.