We introduce an efficient decoder of the color code in $dgeq 2$ dimensions, the Restriction Decoder, which uses any $d$-dimensional toric code decoder combined with a local lifting procedure to find a recovery operation. We prove that the Restriction Decoder successfully corrects errors in the color code if and only if the corresponding toric code decoding succeeds. We also numerically estimate the Restriction Decoder threshold for the color code in two and three dimensions against the bit-filp and phase-flip noise with perfect syndrome extraction. We report that the 2D color code threshold $p_{textrm{2D}} approx 10.2%$ on the square-octagon lattice is on a par with the toric code threshold on the square lattice.