Context. We investigate the stellar population and the origin of diffuse light around brightest cluster galaxies. Aims. We study the stellar population of the dynamically hot stellar halo of NGC 3311, the brightest galaxy in the Hydra I cluster, and that of photometric substructures in the diffuse light to constrain the origin of these components. Methods. We analyze absorption lines in medium-resolution, long-slit spectra in the wavelength range 4800-5800 angstrom obtained with FORS2 at the Very Large Telescope. We measure the equivalent width of Lick indices out to 20 kpc from the center of NGC 3311 and fit them with stellar population models that account for the [alpha/Fe] overabundance. Results. Stars in the dynamically hot halo of NGC 3311 are old (age > 13 Gyr), metal-poor ([Z/H] ~ -0.35), and alpha-enhanced ([alpha/Fe] ~ 0.48). Together with the high velocity dispersion, these measurements indicate that the stars in the halo were accreted from the outskirts of other early-type galaxies, with a possible contribution from dwarf galaxies. We identify a region in the halo of NGC 3311 associated with a photometric substructure where the stellar population is even more metal-poor ([Z/H] ~ -0.73). In this region, our measurements are consistent with a composite stellar population superposed along the line of sight, consisting of stars from the dynamically hot halo of NGC 3311 and stars stripped from dwarf galaxies. The latter component contributes < 28% to the local surface brightness. Conclusions. The build-up of diffuse light around NGC 3311 is on-going. Based on the observed stellar population properties, the dominant part of these stars may have come from the outskirts of bright early-type galaxies, while stars from stripped dwarf galaxies are presently being added.