The detection of binary black hole coalescences by LIGO/Virgo has aroused the interest in primordial black holes (PBHs), because they could be both the progenitors of these black holes and a compelling candidate of dark matter (DM). PBHs are formed soon after the enhanced scalar perturbations re-enter horizon during radiation dominated era, which would inevitably induce gravitational waves as well. Searching for such scalar induced gravitational waves (SIGWs) provides an elegant way to probe PBHs. We perform the first direct search for the signals of SIGWs accompanying the formation of PBHs in North American Nanohertz Observatory for Gravitational waves (NANOGrav) 11-year data set. No statistically significant detection has been made, and hence we place a stringent upper limit on the abundance of PBHs at $95%$ confidence level. In particular, less than one part in a million of the total DM mass could come from PBHs in the mass range of $[2 times 10^{-3}, 7times 10^{-1}] Msun$.