Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userRTApipe, a framework to develop astronomical pipelines for the real-time analysis of scientific data
97
0
0.0
(
0
)
Added by
Nicol\\'o Parmiggiani
Publication date
2021
fields
Physics
and research's language is
English
Ask ChatGPT about the research
No Arabic abstract
In the multi-messenger era, astronomical projects share information about transients phenomena issuing science alerts to the Scientific Community through different communications networks. This coordination is mandatory to understand the nature of these physical phenomena. For this reason, astrophysical projects rely on real-time analysis software pipelines to identify as soon as possible transients (e.g. GRBs), and to speed up external alerts reaction time. These pipelines can share and receive the science alerts through the Gamma-ray Coordinates Network. This work presents a framework designed to simplify the development of real-time scientific analysis pipelines. The framework provides the architecture and the required automatisms to develop a real-time analysis pipeline, allowing the researchers to focus more on the scientific aspects. The framework has been successfully used to develop real-time pipelines for the scientific analysis of the AGILE space mission data. It is planned to reuse this framework for the Super-GRAWITA and AFISS projects. A possible future use for the Cherenkov Telescope Array (CTA) project is under evaluation.
rate research
Read More
Data processing pipelines represent an important slice of the astronomical software library that include chains of processes that transform raw data into valuable information via data reduction and analysis. In this work we present Corral, a Python framework for astronomical pipeline generation. Corral features a Model-View-Controller design pattern on top of an SQL Relational Database capable of handling: custom data models; processing stages; and communication alerts, and also provides automatic quality and structural metrics based on unit testing. The Model-View-Controller provides concept separation between the user logic and the data models, delivering at the same time multi-processing and distributed computing capabilities. Corral represents an improvement over commonly found data processing pipelines in Astronomy since the design pattern eases the programmer from dealing with processing flow and parallelization issues, allowing them to focus on the specific algorithms needed for the successive data transformations and at the same time provides a broad measure of quality over the created pipeline. Corral and working examples of pipelines that use it are available to the community at https://github.com/toros-astro.
The Italian AGILE space mission, with its Gamma-Ray Imaging Detector (GRID) instrument sensitive in the 30 MeV-50 GeV gamma-ray energy band, has been operating since 2007. Agilepy is an open-source Python package to analyse AGILE/GRID data. The package is built on top of the command-line version of the AGILE Science Tools, developed by the AGILE Team, publicly available and released by ASI/SSDC. The primary purpose of the package is to provide an easy to use high-level interface to analyse AGILE/GRID data by simplifying the configuration of the tasks and ensuring straightforward access to the data. The current features are the generation and display of sky maps and light curves, the access to gamma-ray sources catalogues, the analysis to perform spectral model and position fitting, the wavelet analysis. Agilepy also includes an interface tool providing the time evolution of the AGILE off-axis viewing angle for a chosen sky region. The Flare Advocate team also uses the tool to analyse the data during the daily monitoring of the gamma-ray sky. Agilepy (and its dependencies) can be easily installed using Anaconda.
We present a framework to interactively volume-render three-dimensional data cubes using distributed ray-casting and volume bricking over a cluster of workstations powered by one or more graphics processing units (GPUs) and a multi-core CPU. The main design target for this framework is to provide an in-core visualization solution able to provide three-dimensional interactive views of terabyte-sized data cubes. We tested the presented framework using a computing cluster comprising 64 nodes with a total of 128 GPUs. The framework proved to be scalable to render a 204 GB data cube with an average of 30 frames per second. Our performance analyses also compare between using NVIDIA Tesla 1060 and 2050 GPU architectures and the effect of increasing the visualization output resolution on the rendering performance. Although our initial focus, and the examples presented in this work, is volume rendering of spectral data cubes from radio astronomy, we contend that our approach has applicability to other disciplines where close to real-time volume rendering of terabyte-order 3D data sets is a requirement.
In the multi-messenger era, space and ground-based observatories usually develop real-time analysis (RTA) pipelines to rapidly detect transient events and promptly share information with the scientific community to enable follow-up observations. These pipelines can also react to science alerts shared by other observatories through networks such as the Gamma-Ray Coordinates Network (GCN) and the Astronomers Telegram (ATels). AGILE is a space mission launched in 2007 to study X-ray and gamma-ray phenomena. This contribution presents the technologies used to develop two types of AGILE pipelines using the RTApipe framework and an overview of the main scientific results. The first type performs automated analyses on new AGILE data to detect transient events and automatically sends AGILE notices to the GCN network. Since May 2019, this pipeline has sent more than 50 automated notices with a few minutes delay since data arrival. The second type of pipeline reacts to multi-messenger external alerts (neutrinos, gravitational waves, GRBs, and other transients) received through the GCN network and performs hundreds of analyses searching for counterparts in all AGILE instruments data. The AGILE Team uses these pipelines to perform fast follow-up of science alerts reported by other facilities, which resulted in the publishing of several ATels and GCN circulars.
Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each step, a performance monitoring utility needs to be integrated with the pipeline execution. In this work we have developed such a utility and integrated it with the calibration pipeline of the Low Frequency Array, LOFAR, a leading radio telescope. We tested the tool by running the pipeline on several different compute platforms and collected the performance data. Based on this data, we make well informed recommendations on future hardware and software upgrades. The aim of these upgrades is to accelerate the slowest processing steps for this LOFAR pipeline. The pipeline collector suite is open source and will be incorporated in future LOFAR pipelines to create a performance database for all LOFAR processing.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
