We study gravitational lensing of gravitational waves from compact object binaries as a probe of compact dark matter (DM) objects such as primordial black holes. Assuming a point mass lens, we perform parameter estimation of lensed gravitational wave signals from compact object binaries to determine the detectability of the lens with ground based laser interferometers. Then, considering binary populations that LIGO-Virgo has been probing, we derive a constraint on the abundance of compact DM from non-observation of lensed events. We find that the LIGO-Virgo observations imply that compact objects heavier than $M_l = 50M_odot$ can not constitute all DM and less than $15%$ of DM can be in compact objects heavier than $M_l = 200M_odot$. We also show that the DM fraction in compact objects can be probed by LIGO in its final sensitivity for $M_l > 20M_odot$ reaching $0.7%$ of the DM abundance at $M_l > 100M_odot$, and by ET for $M_l > 0.4M_odot$ reaching DM fraction as low as $3times 10^{-5}$ at $M_l > 20M_odot$.