The next generation of space-borne gravitational wave detectors may detect gravitational waves from extreme mass-ratio inspirals with primordial black holes. To produce primordial black holes which contribute a non-negligible abundance of dark matter and are consistent with the observations, a large enhancement in the primordial curvature power spectrum is needed. For a single field slow-roll inflation, the enhancement requires a very flat potential for the inflaton, and this will increase the number of $e$-folds. To avoid the problem, an ultra-slow-roll inflation at the near inflection point is required. We elaborate the conditions to successfully produce primordial black hole dark matter from single field inflation and propose a toy model with polynomial potential to realize the big enhancement of the curvature power spectrum at small scales while maintaining the consistency with the observations at large scales. The power spectrum for the second order gravitational waves generated by the large density perturbations at small scales is consistent with the current pulsar timing array observations.