Here we present a fundamental study on how the ground-state chemical reactivity of a molecule can be modified in a QED scenario, i.e., when it is placed inside a cavity and there is strong coupling between the cavity field and vibrational modes within the molecule. We work with a model system for the molecule (Shin-Metiu model) in which nuclear, electronic and photonic degrees of freedom are treated on the same footing. This simplified model allows the comparison of exact quantum reaction rate calculations with predictions emerging from transition state theory based on the cavity Born-Oppenheimer approach. We demonstrate that QED effects are indeed able to significantly modify activation barriers in chemical reactions and, as a consequence, reaction rates. The critical physical parameter controlling this effect is the permanent dipole of the molecule and how this magnitude changes along the reaction coordinate. We show that the effective coupling can lead to significant single-molecule energy shifts in an experimentally available nanoparticle-on-mirror cavity. We then apply the validated theory to a realistic case (internal rotation in the 1,2-dichloroethane molecule), showing how reactions can be inhibited or catalyzed depending on the profile of the molecular dipole. Furthermore, we discuss the absence of resonance effects in this process, which can be understood through its connection to Casimir-Polder forces. Finally, we treat the case of many-molecule strong coupling, and find collective modifications of reaction rates if the molecular permanent dipole moments are oriented with respected to the cavity field. This demonstrates that collective coupling can also provide a mechanism for modifying ground-state chemical reactivity of an ensemble of molecules coupled to a cavity mode.