We propose a new method to study non-equilibrium dynamics of scalar fields starting from non-Gaussian initial conditions using Keldysh field theory. We use it to study dynamics of phonons coupled to bosonic and fermionic baths, starting from initial Fock states. We find that in one dimension long wavelength phonons coupled to fermionic baths do not thermalize both at low and high bath temperatures. At low temperature, constraints from energy-momentum conservation lead to a narrow bandwidth of particle-hole excitations and the phonons effectively do not feel the effects of this bath. On the other hand, the strong band edge divergence of particle-hole density of states leads to an undamped polariton-like mode of the dressed phonons above the band edge of the particle-hole excitations. These undamped modes contribute to the lack of thermalization of long wavelength phonons at high temperatures. In higher dimensions, these constraints and the divergence of density of states are weakened and leads to thermalization at all wavelengths.