We determine the quantum ground-state properties of ultracold bosonic atoms interacting with the mode of a high-finesse resonator. The atoms are confined by an external optical lattice, whose period is incommensurate with the cavity mode wave length,
and are driven by a transverse laser, which is resonant with the cavity mode. While for pointlike atoms photon scattering into the cavity is suppressed, for sufficiently strong lasers quantum fluctuations can support the build-up of an intracavity field, which in turn amplifies quantum fluctuations. The dynamics is described by a Bose-Hubbard model where the coefficients due to the cavity field depend on the atomic density at all lattice sites. Quantum Monte Carlo simulations and mean-field calculations show that for large parameter regions cavity backaction forces the atoms into clusters with a checkerboard density distribution. Here, the ground state lacks superfluidity and possesses finite compressibility, typical of a Bose-glass. This system constitutes a novel setting where quantum fluctuations give rise to effects usually associated with disorder.
