We compare low velocity impacts that ricochet with the same impact velocity and impact angle into granular media with similar bulk density, porosity and friction coefficient but different mean grain size. The ratio of projectile diameter to mean grain length ranges from 4 in our coarsest medium to 50 in our finest sand. Using high speed video and fluorescent markers, we measure the ratio of pre- to post-impact horizontal and vertical velocity components, which we refer to as coefficients of restitution, and the angle of deflection caused by the impact in the horizontal plane. Coefficients of restitution are sensitive to mean grain size with the ratio associated with the horizontal velocity component about twice as large for our coarsest gravel as that for our finest sand. This implies that coefficients for hydro-static-like, drag-like and lift-like forces, used in empirical force laws, are sensitive to mean grain size. The coefficient that is most strongly sensitive to grain size is the lift coefficient which decreases by a factor of 3 between our coarsest and finest media. The deflection angles are largest in the coarser media and their size approximately depends on grain size to the 3/2 power. This scaling is matched with a model where momentum transfer takes place via collisions with individual grains. The dependence of impact mechanics on substrate size distribution should be considered in future models for populations of objects that impact granular asteroid surfaces.