We present the velocity dispersion measurements of four massive $sim10^{11}M_odot$ quiescent galaxies at $3.2 < z < 3.7$ based on deep H and K$-$band spectra using the Keck/MOSFIRE near-infrared spectrograph. We find high velocity dispersions of order $sigma_esim250$ km/s based on strong Balmer absorption lines and combine these with size measurements based on HST/WFC3 F160W imaging to infer dynamical masses. The velocity dispersion are broadly consistent with the high stellar masses and small sizes. Together with evidence for quiescent stellar populations, the spectra confirm the existence of a population of massive galaxies that formed rapidly and quenched in the early universe $z>4$. Investigating the evolution at constant velocity dispersion between $zsim3.5$ and $zsim2$, we find a large increase in effective radius $0.35pm0.12$ dex and in dynamical-to-stellar mass ratio $<$log(M$_{textrm{dyn}}$/M*)$>$ of 0.33$pm0.08$ dex, with low expected contribution from dark matter. The dynamical masses for our $zsim3.5$ sample are consistent with the stellar masses for a Chabrier initial mass function (IMF), with the ratio $<$log(M$_{textrm{dyn}}$/M$^*_{textrm{Ch}})>$ = -0.13$pm$0.10 dex suggesting an IMF lighter than Salpeter may be common for massive quiescent galaxies at $z>3$. This is surprising in light of the Salpeter or heavier IMFs found for high velocity dispersion galaxies at $zsim2$ and cores of present-day ellipticals, which these galaxies are thought to evolve into. Future imaging and spectroscopic observations with resolved kinematics using the upcoming James Webb Space Telescope could rule out potential systematics from rotation, and confirm these results.