Cd$_3$As$_2$ is one of the prototypical topological Dirac semimetals. Here, we manipulate the band inversion responsible for the emergence of Dirac nodes by alloying Cd$_3$As$_2$ with topologically trivial Zn$_3$As$_2$. We observe the expected topological phase transition around a Zn concentration of $xsim 1$ while the carrier density monotonically decreases as $x$ is increased. For larger $x$, the thermoelectric figure of merit exhibits comparably large values exceeding 0.3 at room temperature, due to the combined effects of a strong enhancement of the thermopower, an only moderate increase of the resistivity, and a suppression of the thermal conductivity. Complementary quantum-oscillation data and optical-conductivity measurements allow to infer that the enhanced thermoelectric performance is due to a flattening of the band structure in the higher-$x$ region in Cd$_{3-x}$Zn$_x$As$_2$.