Topologically nontrivial materials host protected edge states associated with the bulk band inversion through the bulk-edge correspondence. Manipulating such edge states is highly desired for developing new functions and devices practically using their dissipation-less nature and spin-momentum locking. Here we introduce a transition-metal dichalcogenide VTe$_2$, that hosts a charge density wave (CDW) coupled with the band inversion involving V3$d$ and Te5$p$ orbitals. Spin- and angle-resolved photoemission spectroscopy with first-principles calculations reveal the huge anisotropic modification of the bulk electronic structure by the CDW formation, accompanying the selective disappearance of Dirac-type spin-polarized topological surface states that exist in the normal state. Thorough three dimensional investigation of bulk states indicates that the corresponding band inversion at the Brillouin zone boundary dissolves upon CDW formation, by transforming into anomalous flat bands. Our finding provides a new insight to the topological manipulation of matters by utilizing CDWs flexible characters to external stimuli.