We study the elastic scattering time $tau_mathrm{s}$ of ultracold atoms propagating in optical disordered potentials in the strong scattering regime, going beyond the recent work of J. Richard emph{et al.} textit{Phys. Rev. Lett.} textbf{122} 100403 (2019). There, we identified the crossover between the weak and the strong scattering regimes by comparing direct measurements and numerical simulations to the first order Born approximation. Here we focus specifically on the strong scattering regime, where the first order Born approximation is not valid anymore and the scattering time is strongly influenced by the nature of the disorder. To interpret our observations, we connect the scattering time $tau_mathrm{s}$ to the profiles of the spectral functions that we estimate using higher order Born perturbation theory or self-consistent Born approximation. The comparison reveals that self-consistent methods are well suited to describe $tau_mathrm{s}$ for Gaussian-distributed disorder, but fails for laser speckle disorder. For the latter, we show that the peculiar profiles of the spectral functions, as measured independently in V. Volchkov emph{et al.} textit{Phys. Rev. Lett.} textbf{120}, 060404 (2018), must be taken into account. Altogether our study characterizes the validity range of usual theoretical methods to predict the elastic scattering time of matter waves, which is essential for future close comparison between theory and experiments, for instance regarding the ongoing studies on Anderson localization.