The vector vortex coronagraph is an instrument designed for direct detection and spectroscopy of exoplanets over a broad spectral range. Our team is working towards demonstrating contrast performance commensurate with imaging temperate, terrestrial planets orbiting solar-type stars using the High Contrast Imaging Testbed facility at NASAs Jet Propulsion Laboratory. To date, the best broadband performance achieved is $sim$10$^{-8}$ raw contrast over a bandwidth of $Deltalambda/lambda$=10% in the visible regime (central wavelengths of 550-750 nm), while monochromatic tests yield much deeper contrast ($sim$10$^{-9}$ or better). In this study, we analyze the main performance limitations on the testbeds so far, focusing on the quality of the focal plane mask manufacturing. We measure the polarization properties of the masks and the residual electric field in the dark hole as a function of wavelength. Our results suggest that the current performance is limited by localized defects in the in the focal plane masks. A new generation of masks is under test that have fewer defects and promise performance improvements.