By coupling a superconducting weak link to a microwave resonator, recent experiments probed the spectrum and achieved the quantum manipulation of Andreev states in various systems. However, the quantitative understanding of the response of the resonator to changes in the occupancy of the Andreev levels, which are of fermionic nature, is missing. Here, using Bogoliubov-de Gennes formalism to describe the weak link and a general formulation of the coupling to the resonator, we calculate the shift of the resonator frequency as a function of the levels occupancy and describe how transitions are induced by phase or electric field microwave drives. We apply this formalism to analyze recent experimental results obtained using circuit-QED techniques on superconducting atomic contacts and semiconducting nanowire Josephson junctions.