The position-dependent exact-exchange energy per particle $varepsilon_x(z)$ (defined as the interaction between a given electron at $z$ and its exact-exchange hole) at metal surfaces is investigated, by using either jellium slabs or the semi-infinite (SI) jellium model. For jellium slabs, we prove analytically and numerically that in the vacuum region far away from the surface $varepsilon_{x}^{text{Slab}}(z to infty) to - e^{2}/2z$, {it independent} of the bulk electron density, which is exactly half the corresponding exact-exchange potential $V_{x}(z to infty) to - e^2/z$ [Phys. Rev. Lett. {bf 97}, 026802 (2006)] of density-functional theory, as occurs in the case of finite systems. The fitting of $varepsilon_{x}^{text{Slab}}(z)$ to a physically motivated image-like expression is feasible, but the resulting location of the image plane shows strong finite-size oscillations every time a slab discrete energy level becomes occupied. For a semi-infinite jellium, the asymptotic behavior of $varepsilon_{x}^{text{SI}}(z)$ is somehow different. As in the case of jellium slabs $varepsilon_{x}^{text{SI}}(z to infty)$ has an image-like behavior of the form $propto - e^2/z$, but now with a density-dependent coefficient that in general differs from the slab universal coefficient 1/2. Our numerical estimates for this coefficient agree with two previous analytical estimates for the same. For an arbitrary finite thickness of a jellium slab, we find that the asymptotic limits of $varepsilon_{x}^{text{Slab}}(z)$ and $varepsilon_{x}^{text{SI}}(z)$ only coincide in the low-density limit ($r_s to infty$), where the density-dependent coefficient of the semi-infinite jellium approaches the slab {it universal} coefficient 1/2.