We report on the observation of cold collisions between $^6$Li atoms and Yb$^+$ ions. This combination of species has recently been proposed as the most suitable for reaching the quantum limit in hybrid atom-ion systems, due to its large mass ratio. For atoms and ions prepared in the $^2S_{1/2}$ ground state, the charge transfer and association rate is found to be at least~10$^{3}$ times smaller than the Langevin collision rate. These results confirm the excellent prospects of $^6$Li--Yb$^+$ for sympathetic cooling and quantum information applications. For ions prepared in the excited electronic states $^2P_{1/2}$, $^2D_{3/2}$ and $^2F_{7/2}$, we find that the reaction rate is dominated by charge transfer and does not depend on the ionic isotope nor the collision energy in the range $sim$~1--120~mK. The low charge transfer rate for ground state collisions is corroborated by theory, but the $4f$ shell in the Yb$^+$ ion prevents an accurate prediction for the charge transfer rate of the $^2P_{1/2}$, $^2D_{3/2}$ and $^2F_{7/2}$ states. Using textit{ab initio} methods of quantum chemistry we calculate the atom-ion interaction potentials up to energies of 30$times 10^3$~cm$^{-1}$, and use these to give qualitative explanations of the observed rates.