A microscopic derivation of the master equation for the Jaynes-Cummings model with cavity losses is given, taking into account the terms in the dissipator which vary with frequencies of the order of the vacuum Rabi frequency. Our approach allows to single out physical contexts wherein the usual phenomenological dissipator turns out to be fully justified and constitutes an extension of our previous analysis [Scala M. {em et al.} 2007 Phys. Rev. A {bf 75}, 013811], where a microscopic derivation was given in the framework of the Rotating Wave Approximation.