We describe a joint high contrast imaging survey for planets at Keck and VLT of the last large sample of debris disks identified by the Spitzer Space Telescope. No new substellar companions were discovered in our survey of 30 Spitzer-selected targets. We combine our observations with data from four published surveys to place constraints on the frequency of planets around 130 debris disk single stars, the largest sample to date. For a control sample, we assembled contrast curves from several published surveys targeting 277 stars which do not show infrared excesses. We assumed a double power law distribution in mass and semi-major axis of the form f(m,a) = $Cm^{alpha}a^{beta}$, where we adopted power law values and logarithmically flat values for the mass and semi-major axis of planets. We find that the frequency of giant planets with masses 5-20 $M_{rm Jup}$ and separations 10-1000 AU around stars with debris disks is 6.27% (68% confidence interval 3.68 - 9.76%), compared to 0.73% (68% confidence interval 0.20 - 1.80%) for the control sample of stars without disks. These distributions differ at the 88% confidence level, tentatively suggesting distinctness of these samples.