Kondo insulator FeSb$_2$ with large Seebeck coefficient would have potential in thermoelectric applications in cryogenic temperature range if it had not been for large thermal conductivity $kappa$. Here we studied the influence of different chemical substitutions at Fe and Sb site on thermal conductivity and thermoelectric effect in high quality single crystals. At $5%$ of Te doping at Sb site thermal conductivity is suppressed from $sim 250$ W/Km in undoped sample to about 8 W/Km. However, Cr and Co doping at Fe site suppresses thermal conductivity more slowly than Te doping, and even at 20$%$ Cr/Co doping the thermal conductivity remains $sim 30$ W/Km. The analysis of different contributions to phonon scattering indicates that the giant suppression of $kappa$ with Te is due to the enhanced point defect scattering originating from the strain field fluctuations. In contrast, Te-doping has small influence on the correlation effects and then for small Te substitution the large magnitude of the Seebeck coefficient is still preserved, leading to the enhanced thermoelectric figure of merit ($ZTsim 0.05$ at $sim 100$ K) in Fe(Sb$_{0.9}$Te$_{0.1}$)$_2$.