While cavity cooling of a single trapped emitter was demonstrated, cooling of many particles in an array of harmonic traps needs investigation and poses a question of scalability. This work investigates the cooling of a one dimensional atomic array to the ground state of motion via the interaction with the single mode field of a high-finesse cavity. The key factor ensuring the cooling is found to be the mechanical inhomogeneity of the traps. Furthermore it is shown that the pumped cavity mode does not only mediate the cooling but also provides the necessary inhomogeneity if its periodicity differs from the one of the array. This configuration results in the ground state cooling of several tens of atoms within a few milliseconds, a timescale compatible with current experimental conditions. Moreover, the cooling rate scaling with the atom number reveals a drastic change of the dynamics with the size of the array: atoms are either cooled independently, or via collective modes. In the latter case the cavity mediated atom interaction destructively slows down the cooling as well as increases the mean occupation number, quadratically with the atom number. Finally, an order of magnitude speed up of the cooling is predicted as an outcome the optimization scheme based on the adjustment of the array versus the cavity mode periodicity.