Fiber optic communication is the backbone of our modern information society, offering high bandwidth, low loss, weight, size and cost, as well as an immunity to electromagnetic interference. Microwave photonics lends these advantages to electronic sensing and communication systems, but - unlike the field of nonlinear optics - electro-optic devices so far require classical modulation fields whose variance is dominated by electronic or thermal noise rather than quantum fluctuations. Here we present a cavity electro-optic transceiver operating in a millikelvin environment with a mode occupancy as low as 0.025 $pm$ 0.005 noise photons. Our system is based on a lithium niobate whispering gallery mode resonator, resonantly coupled to a superconducting microwave cavity via the Pockels effect. For the highest continuous wave pump power of 1.48 mW we demonstrate bidirectional single-sideband conversion of X band microwave to C band telecom light with a total (internal) efficiency of 0.03 % (0.7 %) and an added output conversion noise of 5.5 photons. The high bandwidth of 10.7 MHz combined with the observed very slow heating rate of 1.1 noise photons s$^{-1}$ puts quantum limited pulsed microwave-optics conversion within reach. The presented device is versatile and compatible with superconducting qubits, which might open the way for fast and deterministic entanglement distribution between microwave and optical fields, for optically mediated remote entanglement of superconducting qubits, and for new multiplexed cryogenic circuit control and readout strategies.