How magnetism emerges in low-dimensional materials such as transition metal dichalcogenides at the monolayer limit is still an open question. Herein, we present a comprehensive study of the magnetic properties of single crystal and monolayer VSe$_{2}$, both experimentally and emph{ab initio}. Magnetometry, X-ray magnetic circular dichrosim (XMCD) and emph{ab initio} calculations demonstrate that the charge density wave in bulk stoichiometric VSe$_{2.0}$ causes a structural distortion with a strong reduction in the density of sates at the Fermi level, prompting the system towards a non-magnetic state but on the verge of a ferromagnetic instability. In the monolayer limit, the structural rearrangement induces a Peierls distortion with the opening of an energy gap at the Fermi level and the absence of magnetic order. Control experiments on defect-induced VSe$_{2-delta}$ single crystals show a breakdown of magnetism, discarding vacancies as a possible origin of magnetic order in VSe$_{2}$.