Inelastic neutron scattering measurements on the molecular dimer-Mott insulator $kappa$-(BEDT-TTF)$_{2}$Cu[N(CN)$_{2}$]Cl reveal a phonon anomaly in a wide temperature range. Starting from $T_{rm ins}sim50$-$60$ K where the charge gap opens, the low-lying optical phonon modes become overdamped upon cooling towards the antiferromagnetic ordering temperature $T_mathrm{N} = 27$ K, where also a ferroelectric ordering at $T_{rm FE} approx T_{rm N}$ occurs. Conversely, the phonon damping becomes small again when spins and charges are ordered below $T_mathrm{N}$, while no change of the lattice symmetry is observed across $T_mathrm{N}$ in neutron diffraction measurements. We assign the phonon anomalies to structural fluctuations coupled to charge and spin degrees of freedom in the BEDT-TTF molecules.