We present an experimentally feasible scheme to implement holonomic quantum computation in the ultrastrong-coupling regime of light-matter interaction. The large anharmonicity and the Z2 symmetry of the quantum Rabi model allow us to build an effective three-level {Lambda}-structured artificial atom for quantum computation. The proposed physical implementation includes two gradiometric flux qubits and two microwave resonators where single-qubit gates are realized by a two-tone driving on one physical qubit, and a two-qubit gate is achieved with a time-dependent coupling between the field quadratures of both resonators. Our work paves the way for scalable holonomic quantum computation in ultrastrongly coupled systems.