The dual-fermion approach provides a formally exact prescription for calculating properties of a correlated electron system in terms of a diagrammatic expansion around dynamical mean-field theory (DMFT). Most practical implementations, however, neglect higher-order interaction vertices beyond two-particle scattering in the dual effective action and further truncate the diagrammatic expansion in the two-particle scattering vertex to a leading-order or ladder-type approximation. In this work we compute the dual-fermion expansion for the two-dimensional Hubbard model including all diagram topologies with two-particle interactions to high orders by means of a stochastic diagrammatic Monte Carlo algorithm. We benchmark the obtained self-energy against numerically exact Diagrammatic Determinant Monte Carlo simulations to systematically assess convergence of the dual-fermion series and the validity of these approximations. We observe that, from high temperatures down to the vicinity of the DMFT Neel transition, the dual-fermion series converges very quickly to the exact solution in the whole range of Hubbard interactions considered ($4 leq U/t leq 12$), implying that contributions from higher-order vertices are small. As the temperature is lowered further, we observe slower series convergence, convergence to incorrect solutions, and ultimately divergence. This happens in a regime where magnetic correlations become significant. We find however that the self-consistent particle-hole ladder approximation yields reasonable and often even highly accurate results in this regime.