Peculiar velocities are a precious tool to study the large-scale distribution of matter in the local universe and test cosmological models. However, present measurements of peculiar velocities are based on empirical distance indicators, which introduce large error bars. Here we present a new method to measure the peculiar velocities, by directly estimating luminosity distances through waveform signals from inspiralling compact binaries and measuring redshifts from electromagnetic (EM) counterparts. In the future, with the distance uncertainty of GW events reducing to $0.1$ per cent by future GW detectors, the uncertainty of the peculiar velocity can be reduced to $10$ km/s at 100 mega parsecs. We find that dozens of GW events with EM counterparts can provide a Hubble constant $H_0$ uncertainty of $0.5%$ and the growth rate of structure with a $0.6%$ precision in the third-generation ground-base GW detectors, which can reconcile the $H_0$ tension and determine the origins for cosmic accelerated expansion.