Thermoelectric materials can be used to convert heat to electric power through the Seebeck effect. We study magneto-thermoelectric figure of merit (ZT) in three-dimensional Dirac semimetal Cd$_3$As$_2$ crystal. It is found that enhancement of power factor and reduction of thermal conductivity can be realized at the same time through magnetic field although magnetoresistivity is greatly increased. ZT can be highly enhanced from 0.17 to 1.1 by more than six times around 350 K under a perpendicular magnetic field of 7 Tesla. The huge enhancement of ZT by magnetic field arises from the linear Dirac band with large Fermi velocity and the large electric thermal conductivity in Cd$_3$As$_2$. Our work paves a new way to greatly enhance the thermoelectric performance in the quantum topological materials.