Three-dimensional string models with half-maximal supersymmetry are believed to be invariant under a large U-duality group which unifies the S and T dualities in four dimensions. We propose an exact, U-duality invariant formula for four-derivative scalar couplings of the form $F(Phi) ( ablaPhi)^4$ in a class of string vacua known as CHL $mathbb{Z}_N$ heterotic orbifolds with $N$ prime, generalizing our previous work which dealt with the case of heterotic string on $T^6$. We derive the Ward identities that $F(Phi)$ must satisfy, and check that our formula obeys them. We analyze the weak coupling expansion of $F(Phi)$, and show that it reproduces the correct tree-level and one-loop contributions, plus an infinite series of non-perturbative contributions. Similarly, the large radius expansion reproduces the exact $F^4$ coupling in four dimensions, including both supersymmetric invariants, plus infinite series of instanton corrections from half-BPS dyons winding around the large circle, and from Taub-NUT instantons. The summation measure for dyonic instantons agrees with the helicity supertrace for half-BPS dyons in 4 dimensions in all charge sectors. In the process we clarify several subtleties about CHL models in $D=4$ and $D=3$, in particular we obtain the exact helicity supertraces for 1/2-BPS dyonic states in all duality orbits.