Making a which-way measurement (WWM) to identify which slit a particle goes through in a double-slit apparatus will reduce the visibility of interference fringes. There has been a long-standing controversy over whether this can be attributed to an uncontrollable momentum transfer. To date, no experiment has characterised the momentum change in a way that relates quantitatively to the loss of visibility. Here, by reconstructing the Bohmian trajectories of single photons, we experimentally obtain the distribution of momentum change, which is observed to be not a momentum kick that occurs at the point of the WWM, but nonclassically accumulates during the propagation of the photons. We further confirm a quantitative relation between the loss of visibility consequent on a WWM and the total (late-time) momentum disturbance. Our results emphasize the role of the Bohmian momentum in giving an intuitive picture of wave-particle duality and complementarity.