Recent experimental advances enable the manipulation of quantum matter by exploiting the quantum nature of light. However, paradigmatic exactly solvable models, such as the Dicke, Rabi or Jaynes-Cummings models for quantum-optical systems, are scarce in the corresponding solid-state, quantum materials context. Focusing on the long-wavelength limit for the light, here, we provide such an exactly solvable model given by a tight-binding chain coupled to a single cavity mode via a quantized version of the Peierls substitution. We show that perturbative expansions in the light-matter coupling have to be taken with care and can easily lead to a false superradiant phase. Furthermore, we provide an analytical expression for the groundstate in the thermodynamic limit, in which the cavity photons are squeezed by the light-matter coupling. In addition, we derive analytical expressions for the electronic single-particle spectral function and optical conductivity. We unveil quantum Floquet engineering signatures in these dynamical response functions, such as analogs to dynamical localization and replica side bands, complementing paradigmatic classical Floquet engineering results. Strikingly, the Drude weight in the optical conductivity of the electrons is partially suppressed by the presence of a single cavity mode through an induced electron-electron interaction.