The partial (up to 7 %) substitution of Cd for Zn in the Yb-based heavy-fermion material YbFe$_2$Zn$_{20}$ is known to induce a slight ($sim 20$ %) reduction of the Sommerfeld specific heat coefficient $gamma$ and a huge (up to two orders of magnitude) reduction of the $T^2$ resistivity coefficient $A$, corresponding to a drastic and unexpected reduction of the Kadowaki-Woods ratio $A/gamma ^2$. Here, Yb $L_{3}$-edge X-ray absorption spectroscopy shows that the Yb valence state is close to $3+$ for all $x$, whereas X-ray diffraction reveals that Cd replace the Zn ions only at the $16c$ site of the $Fdbar{3}m$ cubic structure, leaving the $48f$ and $96g$ sites with full Zn occupation. Ab-initio electronic structure calculations in pure and Cd-doped materials, carried out without considering correlations, show multiple conduction bands with only minor modifications of the band dispersions near the Fermi level and therefore do not explain the resistivity drop introduced by Cd substitution. We propose that the site-selective Cd substitution introduces light conduction bands with substantial contribution of Cd($16c$) $5p$ levels that have weak coupling to the Yb$^{3+}$ $4f$ moments. These light fermions coexist with heavy fermions originated from other conduction bands with larger participation of Zn($48f$ and $96g$) $4p$ levels that remain strongly coupled with the Yb$^{3+}$ local moments.