Ellerman bombs are regions with enhanced Balmer line wing emission and mark magnetic reconnection in the deep solar atmosphere in active regions and quiet Sun. They are often found in regions where opposite magnetic polarities are in close proximity. Recent high resolution observations suggest that Ellerman bombs are more prevalent than thought before. We aim to determine the occurrence of Ellerman bombs in the penumbra of sunspots. We analyze high spatial resolution observations of sunspots in the Balmer H-alpha and H-beta lines as well as auxiliary continuum channels obtained with the Swedish 1-m Solar Telescope and apply the k-means clustering technique to systematically detect and characterize Ellerman Bombs. Features with all the defining characteristics of Ellerman bombs are found in large numbers over the entire penumbra. The true prevalence of these events is only fully appreciated in the H-beta line due to highest spatial resolution and lower chromospheric opacity. We find that the penumbra hosts some of the highest Ellerman bomb densities, only surpassed by the moat in the immediate surroundings of the sunspot. Some penumbral Ellerman bombs show flame morphology and rapid dynamical evolution. Many penumbral Ellerman bombs are fast moving with typical speed of 3.7 km/s and sometimes more than 10 km/s. Many penumbral Ellerman bombs migrate from the inner to the outer penumbra over hundreds of km and some continue moving beyond the outer penumbral boundary into the moat. Many penumbral Ellerman bombs are found in the vicinity of regions with opposite magnetic polarity. We conclude that reconnection is a near continuous process in the low atmosphere of the penumbra of sunspots as manifest in the form of penumbral Ellerman bombs. These are so prevalent that they may be a major sink of sunspot magnetic energy.