We report on a systematic study of temporal Kerr cavity soliton dynamics in the presence of pulsed or amplitude modulated driving fields. In stark contrast to the more extensively studied case of phase modulations, we find that Kerr cavity solitons are not always attracted to maxima or minima of driving field amplitude inhomogeneities. Instead, we find that the solitons are attracted to temporal positions associated with specific driving field values that depend only on the cavity detuning. We describe our findings in light of a spontaneous symmetry breaking instability that physically ensues from a competition between coherent driving and nonlinear propagation effects. In addition to identifying a new type of Kerr cavity soliton behaviour, our results provide valuable insights to practical cavity configurations employing pulsed or amplitude modulated driving fields.