The origin of the monoclinic distortion and domain formation in the quasi two-dimensional layer compound NbTe$_2$ is investigated. Angle-resolved photoemission shows that the Fermi surface is pseudogapped over large portions of the Brillouin zone. Ab initio calculation of the electron and phonon bandstructure as well as the static RPA susceptibility lead us to conclude that Fermi surface nesting and electron-phonon coupling play a key role in the lowering of the crystal symmetry and in the formation of the charge density wave phase.