Beam quality optimization in mammography traditionally considers detection of a target obscured by quantum noise on a homogenous background. It can be argued that this scheme does not correspond well to the clinical imaging task because real mammographic images contain a complex superposition of anatomical structures, resulting in anatomical noise that may dominate over quantum noise. Using a newly developed spectral mammography system, we measured the correlation and magnitude of the anatomical noise in a set of mammograms. The results from these measurements were used as input to an observer-model optimization that included quantum noise as well as anatomical noise. We found that, within this framework, the detectability of tumors and microcalcifications behaved very differently with respect to beam quality and dose. The results for small microcalcifications were similar to what traditional optimization methods would yield, which is to be expected since quantum noise dominates over anatomical noise at high spatial frequencies. For larger tumors, however, low-frequency anatomical noise was the limiting factor. Because anatomical structure has similar energy dependence as tumor contrast, optimal x-ray energy was significantly higher and the useful energy region wider than traditional methods suggest. Measurements on a tissue phantom confirmed these theoretical results. Furthermore, since quantum noise constitutes only a small fraction of the noise, the dose could be reduced substantially without sacrificing tumor detectability. Exposure settings used clinically are therefore not necessarily optimal for this imaging task. The impact of these findings on the mammographic imaging task as a whole is, however, at this stage unclear.