Axion-like particles are dark matter candidates motivated by the Peccei-Quinn mechanism and also occur in effective field theories where their masses and photon couplings are independent. We estimate the dispersion of circularly polarized photons in a background of oscillating axion-like particles (ALPs) with the standard $g_{agamma},a,F_{mu u}tilde F^{mu u}/4$ coupling to photons. This leads to birefringence or rotation of linear polarization by ALP dark matter. Cosmic microwave background (CMB) birefringence limits $Delta alpha lesssim (1.0)^circ$ enable us to constrain the axion-photon coupling $g_{agamma} lesssim 10^{-17}-10^{-12},{rm GeV}^{-1}$, for ultra-light ALP masses $m_a sim 10^{-27} - 10^{-24}$ eV. This improves upon previous axion-photon coupling limits by up to four orders of magnitude. Future CMB observations could tighten limits by another one to two orders.