Error correcting codes with a universal set of transversal gates are the desiderata of realising quantum computing. Such codes, however, are ruled out by the Eastin-Knill theorem. Moreover, it also rules out codes which are covariant with respect to
the action of transversal unitary operations forming continuous symmetries. In this work, starting from an arbitrary code, we construct approximate codes which are covariant with respect to local $SU(d)$ symmetries using quantum reference frames. We show that our codes are capable of efficiently correcting different types of erasure errors. When only a small fraction of the $n$ qudits upon which the code is built are erased, our covariant code has an error that scales as $1/n^2$, which is reminiscent of the Heisenberg limit of quantum metrology. When every qudit has a chance of being erased, our covariant code has an error that scales as $1/n$. We show that the error scaling is optimal in both cases. Our approach has implications for fault-tolerant quantum computing, reference frame error correction, and the AdS-CFT duality.
