Fluctuations in the initial state of heavy-ion collisions are larger at RHIC energy than at LHC energy. This fact can be inferred from recent measurements of the fluctuations of the particle multiplicities and of elliptic flow performed at the two different energies. We show that an analytical description of the initial energy-density field and its fluctuations motivated by the color glass condensate (CGC) effective theory predicts and quantitatively captures the measured energy evolution of these observables. The crucial feature is that fluctuations in the CGC scale like the inverse of the saturation scale of the nuclei.