We perform the covariant canonical quantization of the CPT- and Lorentz-symmetry-violating photon sector of the minimal Standard-Model Extension, which contains a general (timelike, lightlike, or spacelike) fixed background tensor $k_{AF}^mu$. Well-known stability issues, arising from complex-valued energy states, are solved by introducing a small photon mass, orders of magnitude below current experimental bounds. We explicitly construct a covariant basis of polarization vectors, in which the photon field can be expanded. We proceed to derive the Feynman propagator and show that the theory is microcausal. Despite the occurrence of negative energies and vacuum-Cherenkov radiation, we do not find any runaway stability issues, because the energy remains bounded from below. An important observation is that the ordering of the roots of the dispersion relations is the same in any observer frame, which allows for a frame-independent condition that selects the correct branch of the dispersion relation. This turns out to be critical for the consistency of the quantization. To our knowledge, this is the first system for which quantization has consistently been performed, in spite of the fact that the theory contains negative energies in some observer frames.