We study the time coherence of the photoluminescence radiated by spatially indirect excitons confined in a 10 $mu$m electrostatic trap. Above a critical temperature of 1 Kelvin, we show that the photoluminescence has a homogeneous spectral width of about 500 $mu$eV which weakly varies with the exciton density. By contrast, the spectral width reduces by two-fold below the critical temperature and for experimental parameters at which excitons undergo a gray Bose-Einstein condensation. In this regime, we find evidence showing that the excitons temporal coherence is limited by their interaction with a low-concentration of residual excess charges, leading to a minimum photoluminescence spectral width of around 300 $mu$eV.