We study planetesimal evolution in circumbinary disks, focusing on the three systems Kepler 16, 34 and 35 where planets have been discovered recently. We show that for circumbinary planetesimals, in addition to secular forcing, eccentricities evolve on a dynamical timescale, which leads to orbital crossings even in the presence of gas drag. This makes the current locations of the circumbinary Kepler planets hostile to planetesimal accretion. We then present results from simulations including planetesimal formation and dust accretion, and show that even in the most favourable case of 100% efficient dust accretion, in situ growth starting from planetesimals smaller than ~10 km is difficult for Kepler 16b, Kepler 34b and Kepler 35b. These planets were likely assembled further out in the disk, and migrated inward to their current location.