Scaling pair count to next galaxy surveys

The counting of pairs of galaxies or stars according to their distance is at the core of all real-space correlation analyzes performed in astrophysics and cosmology. The next stage upcoming ground (LSST) and space (Euclid) surveys will measure properties of billions of galaxies and tomographic shells will contain hundreds of millions of objects. The combinatorics of the pair count challenges our ability to perform such counting in a minute-scale time which is the order of magnitude useful for optimizing analyses through the intensive use of simulations. The problem is not CPU intensive and is only limited by an efficient access to the data, hence it belongs to the big data category. We use the popular Apache Spark framework to address it and design an efficient high-throughput algorithm to deal with hundreds of millions to billions of input data. To optimize it, we revisit the question of nonhierarchical sphere pixelization based on cube symmetries and develop a new one that we call the Similar Radius Sphere Pixelization (SARSPix) with square-like pixels. It provides the most adapted sphere packing for all distance-related computations. Using LSST-like fast simulations, we compute autocorrelation functions on tomographic bins containing between a hundred million to one billion data points. In all cases we achieve the full construction of a classical pair-distance histogram in about 2 minutes, using a moderate number of worker nodes (16 to 64). This is typically two orders of magnitude higher than what is achieved today and shows the potential of using these new techniques in the field of astronomy on ever-growing datasets. The method presented here is flexible enough to be adapted to any medium size cluster and the software is publicly available from https://github.com/LSSTDESC/SparkCorr.