We present multi-frequency (1-8 GHz) VLA data, combined with VIMOS IFU data and HST imaging, of a z=0.085 radio-quiet type 2 quasar (with L(1.4GHz)~5e23 W/Hz and L(AGN)~2e45 erg/s). Due to the morphology of its emission-line region, the target (J1430+1339) has been referred to as the Teacup AGN in the literature. We identify bubbles of radio emission that are extended ~10-12 kpc to both the east and west of the nucleus. The edge of the brighter eastern bubble is co-spatial with an arc of luminous ionized gas. We also show that the Teacup AGN hosts a compact radio structure, located ~0.8 kpc from the core position, at the base of the eastern bubble. This radio structure is co-spatial with an ionized outflow with an observed velocity of v=-740 km/s. This is likely to correspond to a jet, or possibly a quasar wind, interacting with the interstellar medium at this position. The large-scale radio bubbles appear to be inflated by the central AGN, which indicates that the AGN can also interact with the gas on >~10 kpc scales. Our study highlights that even when a quasar is formally radio-quiet the radio emission can be extremely effective for observing the effects of AGN feedback.