Chemically-active droplets exhibit complex avoiding trajectories. While heterogeneity is inevitable in active matter experiments, it is mostly overlooked in their modelling. Exploiting its geometric simplicity, we fully-resolve the head-on collision of two swimming droplets of different radii and demonstrate that even a small contrast in size critically conditions their collision and subsequent dynamics. We identify three fundamentally-different regimes. The resulting high sensitivity of pairwise collisions is expected to profoundly affect their collective dynamics.