We study three-point functions of operators on the $1/2$ BPS Wilson loop in planar $mathcal{N}=4$ super Yang-Mills theory. The operators we consider are defect changing operators, which change the scalar coupled to the Wilson loop. We first perform the computation at two loops in general set-ups, and then study a special scaling limit called the ladders limit, in which the spectrum is known to be described by a quantum mechanics with the SL(2,$mathbb{R}$) symmetry. In this limit, we resum the Feynman diagrams using the Schwinger-Dyson equation and determine the structure constants at all order in the rescaled coupling constant. Besides providing an interesting solvable example of defect conformal field theories, our result gives invaluable data for the integrability-based approach to the structure constants.