Core-collapse supernova (SN) ejecta are probably structured on both small and large scales, with greater deviations from spherical symmetry nearer the explosion site. Here, we present 2D and 3D gray radiation-hydrodynamics simulations of type II SN light curves from red (RSG) and blue supergiant (BSG) star explosions to investigate the impact on SN observables of inhomogeneities in density or composition, with a characteristic scale set to a few percent of the local radius. Clumping is found to hasten the release of stored radiation, boosting the early time luminosity and shortening the photospheric phase. Around the photosphere, radiation leaks between the clumps where the photon mean free path is greater. Since radiation is stored uniformly in volume, a greater clumping can increase this leakage by storing more and more mass into smaller and denser clumps containing less and less radiation energy. An inhomogeneous medium in which different regions recombine at different temperatures can also impact the light curve. Clumping can thus be a source of diversity in SN brightness. Clumping may lead to a systematic underestimate of ejecta masses from light curve modeling, although a significant offset seems to require a large density contrast of a few tens between clumps and interclump medium.