This paper investigates the electromagnetic production of lepton pairs with low transverse momentum in relativistic heavy ion collisions. We estimate the initial photons transverse momentum contributions by employing models where the average transverse momentum squared of the incoming photon can be calculated in the equivalent photon approximation. We further derive an all order QED resummation for the soft photon radiation, which gives an excellent description of the ATLAS data in ultra-peripheral collisions at the LHC. For peripheral and central collisions, additional $p_T$-broadening effects from multiple interaction with the medium and the magnetic field contributions from the quark-gluon plasma are also discussed.