We derive lower bounds on the density of sources of ultra-high energy cosmic rays from the lack of significant clustering in the arrival directions of the highest energy events detected at the Pierre Auger Observatory. The density of uniformly distributed sources of equal intrinsic intensity was found to be larger than $sim (0.06 - 5) times 10^{-4}$ Mpc$^{-3}$ at 95% CL, depending on the magnitude of the magnetic deflections. Similar bounds, in the range $(0.2 - 7) times 10^{-4}$ Mpc$^{-3}$, were obtained for sources following the local matter distribution.