Carbon nanotubes (CNTs) have recently attracted attention as materials for flexible thermoelectric devices. To provide theoretical guideline of how defects influence the thermoelectric performance of CNTs, we theoretically studied the effects of defects (vacancies and Stone-Wales defects) on its thermoelectric properties; thermal conductance, electrical conductance, and Seebeck coefficient. The results revealed that the defects mostly strongly suppresses the electron conductance, and deteriorates the thermoelectric performance of a CNT. By plugging in the results and the intertube-junction properties into the network model, we further show that the defects with realistic concentrations can significantly degrade the thermoelectric performance of CNT-based networks. Our findings indicate the importance of the purification of CNTs for improving CNT-based thermoelectrics.