Formation of primordial black holes from inflationary fluctuations is accompanied by a scalar induced gravitational wave background. We perform a Bayesian search of such background in the data from Advanced LIGO and Virgos first, second and third observing runs, parametrizing the peak in the curvature power spectrum by a log-normal distribution. The search shows no evidence for such a background. We place 95% confidence level upper limits on the integrated power of the curvature power spectrum peak which, for a narrow width, reaches down to $0.02$ at $10^{17},{rm Mpc}^{-1}$. The resulting constraints are stronger than those arising from BBN or CMB observations. In addition, we find that LIGO and Virgo, at its design sensitivity, and the Einstein Telescope can compete with the constraints related to the abundance of the formed primordial black holes.