The virial expansion characterizes the high-temperature approach to the quantum-classical crossover in any quantum many-body system. Here, we calculate the virial coefficients up to the fifth-order of Fermi gases in 1D, 2D, and 3D, with attractive contact interactions, as relevant for a variety of applications in atomic and nuclear physics. To that end, we discretize the imaginary-time direction and calculate the relevant canonical partition functions. In coarse discretizations, we obtain analytic results featuring relationships between the interaction-induced changes $Delta b_3$, $Delta b_4$, and $Delta b_5$ as functions of $Delta b_2$, the latter being exactly known in many cases by virtue of the Beth-Uhlenbeck formula. Using automated-algebra methods, we push our calculations to progressively finer discretizations and extrapolate to the continuous-time limit. We find excellent agreement for $Delta b_3$ with previous calculations in all dimensions and we formulate predictions for $Delta b_4$ and $Delta b_5$ in 1D and 2D. We also provide, for a range of couplings,the subspace contributions $Delta b_{31}$, $Delta b_{22}$, $Delta b_{41}$, and $Delta b_{32}$, which determine the equation of state and static response of polarized systems at high temperature. As a performance check, we compare the density equation of state and Tan contact with quantum Monte Carlo calculations, diagrammatic approaches, and experimental data where available. Finally, we apply Pade and Pade-Borel resummation methods to extend the usefulness of the virial coefficients to approach and in some cases go beyond the unit-fugacity point.