We report new limits on ultralight scalar dark matter (DM) with dilaton-like couplings to photons that can induce oscillations in the fine-structure constant alpha. Atomic dysprosium exhibits an electronic structure with two nearly degenerate levels
whose energy splitting is sensitive to changes in alpha. Spectroscopy data for two isotopes of dysprosium over a two-year span is analyzed for coherent oscillations with angular frequencies below 1 rad/s. No signal consistent with a DM coupling is identified, leading to new constraints on dilaton-like photon couplings over a wide mass range. Under the assumption that the scalar field comprises all of the DM, our limits on the coupling exceed those from equivalence-principle tests by up to 4 orders of magnitude for masses below 3 * 10^-18 eV. Excess oscillatory power, inconsistent with fine-structure variation, is detected in a control channel, and is likely due to a systematic effect. Our atomic spectroscopy limits on DM are the first of their kind, and leave substantial room for improvement with state-of-the-art atomic clocks.
