A precise link is derived between scalar-graviton S-matrix elements and expectation values of operators in a worldline quantum field theory (WQFT), both used to describe classical scattering of a pair of black holes. The link is formally provided by a worldline path integral representation of the graviton-dressed scalar propagator, which may be inserted into a traditional definition of the S-matrix in terms of time-ordered correlators. To calculate expectation values in the WQFT a new set of Feynman rules is introduced which treats the gravitational field $h_{mu u}(x)$ and position $x_i^mu(tau_i)$ of each black hole on equal footing. Using these both the next-order classical gravitational radiation $langle h^{mu u}(k)rangle$ (previously unknown) and deflection $Delta p_i^mu$ from a binary black hole scattering event are obtained. The latter can also be obtained from the eikonal phase of a $2to2$ scalar S-matrix, which we show to correspond to the free energy of the WQFT.