We investigate the sympathetic relaxation of a free-standing, vibrating carbon nano-tube that is mounted on an atom chip and is immersed in a cloud of ultra-cold atoms. Gas atoms colliding with the nano-tube excite phonons via a Casimir-Polder potential. We use Fermis Golden Rule to estimate the relaxation rates for relevant experimental parameters and develop a fully dynamic theory of relaxation for the multi-mode phononic field embedded in a thermal atomic reservoir. Based on currently available experimental data, we identify the relaxation rates as a function of atom density and temperature that are required for sympathetic ground state cooling of carbon nano-tubes.