The vanadates VO$_2$ and V$_2$O$_3$ are prototypical examples of strongly correlated materials that exhibit a metal-insulator transition. While the phase transitions in these materials have been studied extensively, there is a limited understanding of how the properties of these materials are affected by the presence of defects and doping. In this study we investigate the impact of native point defects in the form of Frenkel defects on the structural, magnetic and electronic properties of VO$_2$ and V$_2$O$_3$, using first-principles calculations. In VO$_2$ the vanadium Frenkel pairs lead to a non-trivial insulating state. The unpaired vanadium interstitial bonds to a single dimer, which leads to a trimer that has one singlet state and one localized single-electron $S=1/2$ state. The unpaired broken dimer created by the vanadium vacancy also has a localized $S=1/2$ state. Thus, the insulating state is created by the singlet dimers, the trimer and the two localized $S=1/2$ states. Oxygen Frenkel pairs, on the other hand, lead to a metallic state in VO$_2$, but are expected to be present in much lower concentrations. In contrast, the Frenkel defects in V$_2$O$_3$ do not directly suppress the insulating character of the material. However, the disorder created by defects in V$_2$O$_3$ alters the local magnetic moments and in turn reduces the energy cost of a transition between the insulating and conducting phases of the material. We also find self-trapped small polarons in V$_2$O$_3$, which has implications for transport properties in the insulating phase.