We construct perturbative classical solutions of the Yang-Mills equations coupled to dynamical point particles carrying color charge. By applying a set of color to kinematics replacement rules first introduced by Bern, Carrasco and Johansson (BCJ), these are shown to generate solutions of d-dimensional dilaton gravity, which we also explicitly construct. Agreement between the gravity result and the gauge theory double copy implies a correspondence between non-Abelian particles and gravitating sources with dilaton charge. When the color sources are highly relativistic, dilaton exchange decouples, and the solutions we obtain match those of pure gravity. We comment on possible implications of our findings to the calculation of gravitational waveforms in astrophysical black hole collisions, directly from computationally simpler gluon radiation in Yang-Mills theory.