We present spectroscopic observations obtained with the infrared Spitzer Space Telescope, which provide insight into the H$_2$ physics and gas energetics in photodissociation Regions (PDRs) of low to moderate far-ultraviolet (FUV) fields and densities. We analyze data on six well known Galactic PDRs (L1721, California, N7023E, Horsehead, rho Oph, N2023N), sampling a poorly explored range of excitation conditions ($chi sim 5-10^3$), relevant to the bulk of molecular clouds in galaxies. Spitzer observations of H$_2$ rotational lines are complemented with H$_2$ data, including ro-vibrational line measurements, obtained with ground-based telescopes and ISO, to constrain the relative contributions of ultraviolet pumping and collisions to the H$_2$ excitation. The data analysis is supported by model calculations with the Meudon PDR code. The observed column densities of rotationally excited H$_2$ are observed to be much higher than PDR model predictions. In the lowest excitation PDRs, the discrepancy between the model and the data is about one order of magnitude for rotational levels $J ge $3. We discuss whether an enhancement in the H$_2$ formation rate or a local increase in photoelectric heating, as proposed for brighter PDRs in former ISO studies, may improve the data-model comparison. We find that an enhancement in the H$_2$ formation rates reduces the discrepancy, but the models still fall short of the data. This large disagreement suggests that our understanding of the formation and excitation of H$_2$ and/or of PDRs energetics is still incomplete. We discuss several explanations, which could be further tested using the Herschel Space Telescope