We study the phononic collective modes of the pairing field $Delta$ and their corresponding signature in both the order-parameter and density response functions for a superfluid Fermi gas at all temperatures below $T_c$ in the collisionless regime. The spectra of collective modes are calculated within the Gaussian Pair Fluctuation approximation. We deal with the coupling of these modes to the fermionic continuum of quasiparticle-quasihole excitations by performing a non-perturbative analytic continuation of the pairing field propagator. At low temperature, we recover the known exponential temperature dependence of the damping rate and velocity shift of the Anderson-Bogoliubov branch. In the vicinity of $T_c$, we find analytically a weakly-damped collective mode whose velocity vanishes with a critical exponent of $1/2$, and whose quality factor diverges logarithmically with $T_c-T$, thereby clarifying an existing debate in the literature (Andrianov et al. Th. Math. Phys. 28, 829, Ohashi et al. J. Phys. Jap. 66, 2437). A transition between these two phononic branches is visible at intermediary temperatures, particularly in the BCS limit where the phase-phase response function displays two maxima.