The Dark Sky Simulations are an ongoing series of cosmological N-body simulations designed to provide a quantitative and accessible model of the evolution of the large-scale Universe. Such models are essential for many aspects of the study of dark ma
tter and dark energy, since we lack a sufficiently accurate analytic model of non-linear gravitational clustering. In July 2014, we made available to the general community our early data release, consisting of over 55 Terabytes of simulation data products, including our largest simulation to date, which used $1.07 times 10^{12}~(10240^3)$ particles in a volume $8h^{-1}mathrm{Gpc}$ across. Our simulations were performed with 2HOT, a purely tree-based adaptive N-body method, running on 200,000 processors of the Titan supercomputer, with data analysis enabled by yt. We provide an overview of the derived halo catalogs, mass function, power spectra and light cone data. We show self-consistency in the mass function and mass power spectrum at the 1% level over a range of more than 1000 in particle mass. We also present a novel method to distribute and access very large datasets, based on an abstraction of the World Wide Web (WWW) as a file system, remote memory-mapped file access semantics, and a space-filling curve index. This method has been implemented for our data release, and provides a means to not only query stored results such as halo catalogs, but also to design and deploy new analysis techniques on large distributed datasets.
The usage of the high-level scripting language Python has enabled new mechanisms for data interrogation, discovery and visualization of scientific data. We present yt, an open source, community-developed astrophysical analysis and visualization toolk
it for data generated by high-performance computing (HPC) simulations of astrophysical phenomena. Through a separation of responsibilities in the underlying Python code, yt allows data generated by incompatible, and sometimes even directly competing, astrophysical simulation platforms to be analyzed in a consistent manner, focusing on physically relevant quantities rather than quantities native to astrophysical simulation codes. We present on its mechanisms for data access, capabilities for MPI-parallel analysis, and its implementation as an in situ analysis and visualization tool.
Modern N-body cosmological simulations contain billions ($10^9$) of dark matter particles. These simulations require hundreds to thousands of gigabytes of memory, and employ hundreds to tens of thousands of processing cores on many compute nodes. In
order to study the distribution of dark matter in a cosmological simulation, the dark matter halos must be identified using a halo finder, which establishes the halo membership of every particle in the simulation. The resources required for halo finding are similar to the requirements for the simulation itself. In particular, simulations have become too extensive to use commonly-employed halo finders, such that the computational requirements to identify halos must now be spread across multiple nodes and cores. Here we present a scalable-parallel halo finding method called Parallel HOP for large-scale cosmological simulation data. Based on the halo finder HOP, it utilizes MPI and domain decomposition to distribute the halo finding workload across multiple compute nodes, enabling analysis of much larger datasets than is possible with the strictly serial or previous parallel implementations of HOP. We provide a reference implementation of this method as a part of the toolkit yt, an analysis toolkit for Adaptive Mesh Refinement (AMR) data that includes complementary analysis modules. Additionally, we discuss a suite of benchmarks that demonstrate that this method scales well up to several hundred tasks and datasets in excess of $2000^3$ particles. The Parallel HOP method and our implementation can be readily applied to any kind of N-body simulation data and is therefore widely applicable.
Simulations of the formation of Population III (Pop III) stars suggest that they were much more massive than the Pop II and Pop I stars observed today. This is due to the collapse dynamics of metal-free gas, which is regulated by the radiative coolin
g of molecular hydrogen. We study how the collapse of gas clouds is altered by the addition of metals to the star-forming environment by performing a series of simulations of pre-enriched star formation at various metallicities. For metallicities below the critical metallicity, Z_cr, collapse proceeds similarly to the metal-free case, and only massive objects form. For metallicities well above Z_cr, efficient cooling rapidly lowers the gas temperature to the temperature of the CMB. The gas is unable to radiatively cool below the CMB temperature, and becomes thermally stable. For high metallicities, Z >= 10^-2.5 Zsun, this occurs early in the evolution of the gas cloud, when the density is still relatively low. The resulting cloud-cores show little or no fragmentation, and would most likely form massive stars. If the metallicity is not vastly above Z_cr, the cloud cools efficiently but does not reach the CMB temperature, and fragmentation into multiple objects occurs. We conclude that there were three distinct modes of star formation at high redshift (z >= 4): a `primordial mode, producing massive stars (10s to 100s Msun) at very low metallicities (Z <= 10^-3.75 Zsun); a CMB-regulated mode, producing moderate mass (10s of Msun) stars at high metallicites (Z >= 10^-2.5 Zsun at redshift z ~ 15-20); and a low-mass (a few Msun) mode existing between those two metallicities. As the universe ages and the CMB temperature decreases, the range of the low mass mode extends to higher metallicities, eventually becoming the only mode of star formation. (Abridged)
Numerous cosmological hydrodynamic studies have addressed the formation of galaxies. Here we choose to study the first stages of galaxy formation, including non-equilibrium atomic primordial gas cooling, gravity and hydrodynamics. Using initial condi
tions appropriate for the concordance cosmological model of structure formation, we perform two adaptive mesh refinement simulations of ~10^8 M_sun galaxies at high redshift. The calculations resolve the Jeans length at all times with more than 16 cells and capture over 14 orders of magnitude in length scales. In both cases, the dense, 10^5 solar mass, one parsec central regions are found to contract rapidly and have turbulent Mach numbers up to 4. Despite the ever decreasing Jeans length of the isothermal gas, we only find one site of fragmentation during the collapse. However, rotational secular bar instabilities transport angular momentum outwards in the central parsec as the gas continues to collapse and lead to multiple nested unstable fragments with decreasing masses down to sub-Jupiter mass scales. Although these numerical experiments neglect star formation and feedback, they clearly highlight the physics of turbulence in gravitationally collapsing gas. The angular momentum segregation seen in our calculations plays an important role in theories that form supermassive black holes from gaseous collapse.
