Using effective field theory methods, we calculate for the first time the complete fourth-order term in the Fermi-momentum or $k_{rm F} a_s$ expansion for the ground-state energy of a dilute Fermi gas. The convergence behavior of the expansion is examined for the case of spin one-half fermions and compared against quantum Monte-Carlo results, showing that the Fermi-momentum expansion is well-converged at this order for $| k_{rm F} a_s | lesssim 0.5$.