The majority of spiral and elliptical galaxies in the Universe host very dense and compact stellar systems at their centres known as nuclear star clusters (NSCs). In this work we study the stellar populations and star formation histories (SFH) of the NSCs of six nearby galaxies with stellar masses ranging between $2$ and $8times10^9~{rm M_{odot}}$ (four late-type spirals and two early-types) with high resolution spectroscopy. Our observations are taken with the X-Shooter spectrograph at the VLT. We make use of an empirical simple stellar population (SSP) model grid to fit composite stellar populations to the data and recover the SFHs of the nuclei. We find that the nuclei of all late-type galaxies experienced a prolonged SFH, while the NSCs of the two early-types are consistent with SSPs. The NSCs in the late-type galaxies sample appear to have formed a significant fraction of their stellar mass already more than $10$ Gyr ago, while the NSCs in the two early-type galaxies are surprisingly younger. Stars younger than $100$ Myr are present in at least two nuclei: NGC 247 and NGC 7793, with some evidence for young star formation in NGC 300s NSC. The NSCs of the spirals NGC 247 and NGC 300 are consistent with prolonged in situ star formation with a gradual metallicity enrichment from $sim-1.5$ dex more than $10$ Gyr ago, reaching super-Solar values few hundred Myr ago. NGC 3621 appears to be very metal rich already in the early Universe and NGC 7793 presents us with a very complex SFH, likely dominated by merging of various massive star clusters coming from different environments.