Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userThe IceProd Framework: Distributed Data Processing for the IceCube Neutrino Observatory
280
0
0.0
(
0
)
Added by
J. C. D\\'iaz-V\\'elez
Publication date
2013
fields
Informatics Engineering
and research's language is
English
Ask ChatGPT about the research
No Arabic abstract
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, iden- tify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. IceProd is a distributed management system based on Python, XML-RPC and GridFTP. It is driven by a central database in order to coordinate and admin- ister production of simulations and processing of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, Condor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.
rate research
Read More
The industry and academia have proposed many distributed graph processing systems. However, the existing systems are not friendly enough for users like data analysts and algorithm engineers. On the one hand, the programing models and interfaces differ a lot in the existing systems, leading to high learning costs and program migration costs. On the other hand, these graph processing systems are tightly bound to the underlying distributed computing platforms, requiring users to be familiar with distributed computing. To improve the usability of distributed graph processing, we propose a unified distributed graph programming framework UniGPS. Firstly, we propose a unified cross-platform graph programming model VCProg for UniGPS. VCProg hides details of distributed computing from users. It is compatible with the popular graph programming models Pregel, GAS, and Push-Pull. VCProg programs can be executed by compatible distributed graph processing systems without modification, reducing the learning overheads of users. Secondly, UniGPS supports Python as the programming language. We propose an interprocess-communication-based execution environment isolation mechanism to enable Java/C++-based systems to call user-defined methods written in Python. The experimental results show that UniGPS enables users to process big graphs beyond the memory capacity of a single machine without sacrificing usability. UniGPS shows near-linear data scalability and machine scalability.
The IceCube Neutrino Observatory opened the window on neutrino astronomy by discovering high-energy astrophysical neutrinos in 2013 and identifying the first compelling astrophysical neutrino source, the blazar TXS0506+056, in 2017. In this talk, we will discuss the science reach and ongoing development of the IceCube-Gen2 facility---a planned extension to IceCube. IceCube-Gen2 will increase the rate of observed cosmic neutrinos by an order of magnitude, be able to detect five-times fainter neutrino sources, and extend the measurement of astrophysical neutrinos several orders of magnitude higher in energy. We will discuss the envisioned design of the instrument, which will include an enlarged in-ice optical array, a surface array for the study of cosmic-rays, and a shallow radio array to detect ultra-high energy (>100 PeV) neutrinos. we will also highlight ongoing efforts to develop and test new instrumentation for IceCube-Gen2.
Papers on research & development towards IceCube-Gen2, the next generation neutrino observatory at South Pole, submitted to the 35th International Cosmic Ray Conference (ICRC 2017, Busan, South Korea) by the IceCube-Gen2 Collaboration.
This paper presents BigDL (a distributed deep learning framework for Apache Spark), which has been used by a variety of users in the industry for building deep learning applications on production big data platforms. It allows deep learning applications to run on the Apache Hadoop/Spark cluster so as to directly process the production data, and as a part of the end-to-end data analysis pipeline for deployment and management. Unlike existing deep learning frameworks, BigDL implements distributed, data parallel training directly on top of the functional compute model (with copy-on-write and coarse-grained operations) of Spark. We also share real-world experience and war stories of users that have adopted BigDL to address their challenges(i.e., how to easily build end-to-end data analysis and deep learning pipelines for their production data).
The past decade has welcomed the emergence of cosmic neutrinos as a new messenger to explore the most extreme environments of the universe. The discovery measurement of cosmic neutrinos, announced by IceCube in 2013, has opened a new window of observation that has already resulted in new fundamental information that holds the potential to answer key questions associated with the high-energy universe, including: what are the sources in the PeV sky and how do they drive particle acceleration; where are cosmic rays of extreme energies produced, and on which paths do they propagate through the universe; and are there signatures of new physics at TeV-PeV energies and above? The planned advancements in neutrino telescope arrays in the next decade, in conjunction with continued progress in broad multimessenger astrophysics, promise to elevate the cosmic neutrino field from the discovery to the precision era and to a survey of the sources in the neutrino sky. The planned detector upgrades to the IceCube Neutrino Observatory, culminating in IceCube-Gen2 (an envisaged $400M facility with anticipated operation in the next decade, described in this white paper) are the cornerstone that will drive the evolution of neutrino astrophysics measurements.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
