We study the temperature dependence of the low energy phonons in the $(H, 0, L)$ reciprocal plane of the highly ordered ortho-II YBa$_2$Cu$_3$O$_{6.55}$ cuprate high temperature superconductor by means of high-resolution inelastic x-ray scattering. Anomalies associated with the emergence of long-range charge density wave (CDW) fluctuations are observed, and are qualitatively similar to those previously observed in the $(0, K, L)$ plane. This confirms the unconventional nature of this bi-dimensional CDW, which is not soft-phonon driven. With the support of first principles calculations, the symmetry of the anomalous phonon is identified and is found to match that of the charge modulation. This suggests in turn that these anomalies originate from a direct coupling between the phonons and the collective CDW excitations.