The non-Markovian collision integral is obtained on the base of the Kadanoff-Baym equations for Green functions in a form with allowance for small retardation effects. The collisional relaxation times and damping width of the giant isovector dipole resonances in nuclear matter are calculated. For an infinite Fermi liquid the dependence of the relaxation times on the collective vibration frequency and the temperature corresponds to the Landaus prescription.