We use the semi-analytical model of galaxy formation GALFORM to characterise an indirect signature of AGN feedback in the environment of radio galaxies at high redshifts. The predicted environment of radio galaxies is denser than that of radio-quiet galaxies with the same stellar mass. This is consistent with observational results from the CARLA survey. Our model shows that the differences in environment are due to radio galaxies being hosted by dark matter haloes that are ~1.5 dex more massive than those hosting radio-quiet galaxies with the same stellar mass. By running a control-simulation in which AGN feedback is switched-off, we identify AGN feedback as the primary mechanism affecting the build-up of the stellar component of radio galaxies, thus explaining the different environment in radio galaxies and their radio-quiet counterparts. The difference in host halo mass between radio loud and radio quiet galaxies translates into different galaxies populating each environment. We predict a higher fraction of passive galaxies around radio loud galaxies compared to their radio-quiet counterparts. Furthermore, such a high fraction of passive galaxies shapes the predicted infrared luminosity function in the environment of radio galaxies in a way that is consistent with observational findings. Our results suggest that the impact of AGN feedback at high redshifts and environmental mechanisms affecting galaxies in high halo masses can be revealed by studying the environment of radio galaxies, thus providing new constraints on galaxy formation physics at high redshifts.