Using a leading-order semiclassical approximation, we calculate the third- and fourth-order virial coefficients of nonrelativistic spin-1/2 fermions in a harmonic trapping potential in arbitrary spatial dimensions, and as functions of temperature, trapping frequency and coupling strength. Our simple, analytic results for the interaction-induced changes $Delta b_3$ and $Delta b_4$ agree qualitatively, and in some regimes quantitatively, with previous numerical calculations for the unitary limit of three-dimensional Fermi gases.