We present new imaging and spectroscopic observations of six ultra-compact dwarf (UCD) galaxies in the Virgo Cluster, along with re-analysed data for five Fornax Cluster UCDs. These are the most luminous UCDs: -14<Mv<-12 mag. Our HST imaging shows that most of the UCDs have shallow or steep cusps in their cores; only one UCD has a flat ``King core. None of the UCDs show tidal cutoffs down to our limiting surface brightness. Spectroscopic analysis shows that Virgo UCDs are older than 8 Gyr and have metallicities in the range [Z/H]=-1.35...+0.35 dex. Five Virgo UCDs have super-solar alpha/Fe abundance ratios typical of old stellar populations found in globular clusters and elliptical galaxies. Virgo UCDs have structural and dynamical properties similar to Fornax UCDs. The Virgo and Fornax UCDs all have masses ~2-9x10^7 Msun and mass-to-light ratios ~3-5 Msun/Lsun,v. The dynamical M/L values for Virgo UCDs are consistent with SSP model predictions: Virgo UCDs do not require dark matter to explain their mass-to-light ratios. We conclude that the internal properties of Virgo UCDs are consistent with them being the high-mass/high-luminosity extreme of known globular cluster populations. We refrain from any firm conclusions on Fornax UCD origins until accurate age, metallicity and alpha-abundance estimates are obtained for them. Some of our results, notably the fundamental plane projections are consistent with the formation of UCDs by the simple removal of the halo from the nuclei of nucleated dwarf galaxies. However the ages, metallicities and abundances for Virgo UCDs are not consistent with this simple stripping model. It might be consistent with more sophisticated models of the stripping process that include the effects of gas removal on the chemical evolution of the nuclei.