We study Landau damping in the 1+1D Vlasov-Poisson system using a Fourier-Hermite spectral representation. We describe the propagation of free energy in phase space using forwards and backwards propagating Hermite modes recently developed for gyrokinetics [Schekochihin et al. (2014)]. The change in the electric field corresponds to the net Hermite flux via a free energy evolution equation. In linear Landau damping, decay in the electric field corresponds to forward propagating Hermite modes; in nonlinear damping, the initial decay is followed by a growth phase characterised by the generation of backwards propagating Hermite modes by the nonlinear term. The free energy content of the backwards propagating modes increases exponentially until balancing that of the forward propagating modes. Thereafter there is no systematic net Hermite flux, so the electric field cannot decay and the nonlinearity effectively suppresses Landau damping. These simulations are performed using the fully-spectral 5D gyrokinetics code SpectroGK [Parker et al. 2014], modified to solve the 1+1D Vlasov-Poisson system. This captures Landau damping via an iterated Lenard-Bernstein collision operator or via Hou-Li filtering in velocity space. Therefore the code is applicable even in regimes where phase-mixing and filamentation are dominant.