We have examined an isovalent Rh substitution effect on the transport properties of the thermoelectric oxide Ca$_3$Co$_{4}$O$_9$ using single-crystalline form. With increasing Rh content $x$, both the electrical resistivity and the Seebeck coefficient change systematically up to $x=0.6$ for Ca$_3$Co$_{4-x}$Rh$_{x}$O$_9$ samples. In the Fermi-liquid regime where the resistivity behaves as $rho=rho_0+AT^2$ around 120 K, the $A$ value decreases with increasing Rh content, indicating that the correlation effect is weakened by Rh $4d$ electrons with extended orbitals. We find that, in contrast to such a weak correlation effect observed in the resistivity of Rh-substituted samples, the low-temperature Seebeck coefficient is increased with increasing Rh content, which is explained with a possible enhancement of a pseudogap associated with the short-range order of spin density wave. In high-temperature range above room temperature, we show that the resistivity is largely suppressed by Rh substitution while the Seebeck coefficient becomes almost temperature-independent, leading to a significant improvement of the power factor in Rh-substituted samples. This result is also discussed in terms of the differences in the orbital size and the associated spin state between Co $3d$ and Rh $4d$ electrons.