We re-examine the thermodynamic properties of the coupled dimer system Cu$_2$(C$_5$H$_{12}$N$_2$)$_2$Cl$_4$ under magnetic field in the light of recent NMR experiments [Clemancey {it et al.}, Phys. Rev. Lett. {bf 97}, 167204 (2006)] suggesting the existence of a finite Dzyaloshinskii-Moriya interaction. We show that including such a spin anisotropy greatly improves the fit of the magnetization curve and gives the correct trend of the insofar unexplained anomalous behavior of the specific heat in magnetic field at low temperature.