We derive an energy-dependent decay-time distribution function from the multi-exponential decay of the ensemble photoluminescence (PL) of InGaN/GaN quantum dots (QDs), which agrees well with recently published single-QD time-resolved PL measurements. Using eight-band k.p modelling, we show that the built-in piezo- and pyroelectric fields within the QDs cause a sensitive dependence of the radiative lifetimes on the exact QD geometry and composition. Moreover, the radiative lifetimes also depend heavily on the composition of the direct surrounding of the QDs. A broad lifetime distribution occurs even for moderate variations of the QD structure. Thus, for unscreened fields a multi-exponential decay of the ensemble PL is generally expected in this material system.