We study the interactions among phonons and the phonon lifetime in a pair-condensed Fermi gas in the BEC-BCS crossover in the collisionless regime. To compute the phonon-phonon coupling amplitudes we use a microscopic model based on a generalized BCS Ansatz including moving pairs, which allows for a systematic expansion around the mean field BCS approximation of the ground state. We show that the quantum hydrodynamic expression of the amplitudes obtained by Landau and Khalatnikov apply only on the energy shell, that is for resonant processes that conserve energy. The microscopic model yields the same excitation spectrum as the Random Phase Approximation, with a linear (phononic) start and a concavity at low wave number that changes from upwards to downwards in the BEC-BCS crossover. When the concavity of the dispersion relation is upwards at low wave number, the leading damping mechanism at low temperature is the Beliaev-Landau process 2 phonons $leftrightarrow$ 1 phonon while, when the concavity is downwards, it is the Landau-Khalatnikov process 2 phonons $leftrightarrow$ 2 phonons. In both cases, by rescaling the wave vectors to absorb the dependence on the interaction strength, we obtain a universal formula for the damping rate. This universal formula corrects and extends the original analytic results of Landau and Khalatnikov [ZhETF {bf 19}, 637 (1949)] for the $2leftrightarrow2$ processes in the downward concavity case. In the upward concavity case, for the Beliaev 1$leftrightarrow$ 2 process for the unitary gas at zero temperature, we calculate the damping rate of an excitation with wave number $q$ including the first correction proportional to $q^7$ to the $q^5$ hydrodynamic prediction, which was never done before in a systematic way.