We demonstrate that a recently proposed classical double copy procedure to construct the effective action of two massive particles in dilaton-gravity from the analogous problem of two color charged particles in Yang-Mills gauge theory fails at next-to-next-to-leading orders in the post-Minkowskian (3PM) or post-Newtonian (2PN) expansions.