We investigate a numerical model for AGN feedback where for the first time a relativistic particle population in AGN-inflated bubbles is followed within a full cosmological context. In our high-resolution simulations of galaxy cluster formation, we assume that BH accretion is accompanied by energy feedback that occurs in two different modes, depending on the accretion rate itself. Unlike in previous work, we inject a non-thermal particle population of relativistic protons into the AGN bubbles, instead of adopting a purely thermal heating. We then follow the subsequent evolution of the cosmic ray (CR) plasma inside the bubbles, considering both its hydrodynamical interactions and dissipation processes relevant for the CR population. Due to the different buoyancy of relativistic plasma and the comparatively long CR dissipation timescale we find substantial changes in the evolution of clusters as a result of CR feedback. In particular, the non-thermal population can provide significant pressure support in central cluster regions at low thermal temperatures, providing a natural explanation for the decreasing temperature profiles found in cool core clusters. At the same time, the morphologies of the bubbles and of the induced X-ray cavities show a striking similarity to observational findings. AGN feedback with CRs also proves efficient in regulating cluster cooling flows so that the total baryon fraction in stars becomes limited to realistic values of the order of 10%. We find that the partial CR support of the intracluster gas also affects the expected signal of the thermal Sunyaev-Zeldovich effect, with typical modifications of the integrated Compton-y parameter within the virial radius of the order of 10%. [Abridged]