A solid with larger sound speeds exhibits higher lattice thermal conductivity (k_{lat}). Diamond is a prominent instance where its mean sound speed is 14400 m s-1 and k_{lat} is 2300 W m-1 K-1. Here, we report an extreme exception that CuP2 has quite large mean sound speeds of 4155 m s-1, comparable to GaAs, but the single crystals show a very low lattice thermal conductivity of about 4 W m-1 K-1 at room temperature, one order of magnitude smaller than GaAs. To understand such a puzzling thermal transport behavior, we have thoroughly investigated the atomic structure and lattice dynamics by combining neutron scattering techniques with first-principles simulations. Cu atoms form dimers sandwiched in between the layered P atomic networks and the dimers vibrate as a rattling mode with frequency around 11 meV. This mode is manifested to be remarkably anharmonic and strongly scatters acoustic phonons to achieve the low k_{lat}. Such a dimer rattling behavior in layered structures might offer an unprecedented strategy for suppressing thermal conduction without involving atomic disorder.