The Lyman-$alpha$ (Ly$alpha$) emission line has been ubiquitously used to confirm and study high redshift galaxies. We report on the line morphology as seen in the 2D spectra from the VIMOS Ultra Deep Survey in a sample of 914 Ly$alpha$ emitters from a parent sample of 4192 star-forming galaxies at $2<z_mathrm{spec}lesssim6$. The study of the spatial extent of Ly$alpha$ emission provides insight into the escape of Ly$alpha$ photons from galaxies. We classify the line emission as either non-existent, coincident, projected spatial offset, or extended with respect to the observed 2D UV continuum emission. The line emitters in our sample are classified as ~45% coincident, ~24% extended and ~11% offset emitters. For galaxies with detected UV continuum, we show that extended Ly$alpha$ emitters (LAEs) correspond to the highest equivalent width galaxies (with an average $W_mathrm{Lyalpha}sim-22${AA}). This means that this class of objects is the most common in narrow-band selected samples, which usually select high equivalent width LAEs, $<-20${AA}. Extended Ly$alpha$ emitters are found to be less massive, less star-forming, with lower dust content, and smaller UV continuum sizes ($r_{50}sim0.9$kpc) of all the classes considered here. We also find that galaxies with larger UV-sizes have lower fractions of Ly$alpha$ emitters. By stacking the spectra per emitter class we find that the weaker Ly$alpha$ emitters have stronger low ionization inter-stellar medium (ISM) absorption lines. Interestingly, we find that galaxies with Ly$alpha$ offset emission (median separation of $1.1_{-0.8}^{+1.3}$kpc from UV continuum) show similar velocity offsets in the ISM as those with no visible emission (and different from other Ly$alpha$ emitting classes). This class of objects may hint at episodes of gas accretion, bright offset clumps, or on-going merging activity into the larger galaxies.