A magnetic vortex occurs as an equilibrium configuration in thin ferromagnetic platelets of micron and sub-micron size and is characterised by an in-plane curling magnetisation. At the centre, a magnetic singularity is avoided by an out-of-plane magnetisation core. This core has a gyrotropic excitation mode, which corresponds to a circular motion of the vortex around its equilibrium position, where the rotation sense is determined by the direction of the vortex core magnetisation, its polarisation. Unlike linear fields or spin polarised currents, which excite both polarisation states, an in-plane rotating field can selectively excite one of the polarisation states. Here we report the observation of vortex dynamics in response to rotating magnetic fields, imaged with time-resolved scanning X-ray microscopy. We demonstrate that the rotating field only excites the gyrotropic mode if the rotation sense of the field coincides with the vortex gyration sense and that such a field can selectively reverse the vortex polarisation.