Arrays of wavelength scale scatterers are a promising platform for designing optical elements with a compact footprint. The large number of degrees of freedom in this system allows for unique and plentiful functionalities. However, the many variables also create a complex design problem. While intuitive forward design methods work for simple optical elements, they often fail to produce complicated elements, especially those involving multiple elements. We present an inverse design methodology for large arrays of wavelength scale spheres based on both adjoint optimization or sensitivity analysis and generalized multi-sphere Mie theory as a solution to the design problem. We validate our methodology by designing two sets of optical elements with scatterers on sub-wavelength and super-wavelength periodicity grids. Both sets consist of a singlet and a doublet lens, with one and two layers of spheres respectively designed for 1550 nm. The designed NA is ~0.33 (~0.5) for the sub-wavelength (super-wavelength) periodic structure. We find that with the sub-wavelength periodicity, the full width at half maximum of the focal spot produced by the singlet and doublet is smaller than that produced by an ideal lens with the same geometric parameters. Finally, we simulate a realistic experimental scenario for the doublet where the spheres are placed on a substrate with the same refractive index. We find the performance is similar, but with lower intensity at the focal spot and larger spot size. The method described here will simplify the design procedure for complicated multifunctional optical elements and or scatterer array-based volume optics based on a specified figure of merit.