Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userBuilding an interoperable, distributed storage and authorization system
139
0
0.0
(
0
)
Ask ChatGPT about the research
No Arabic abstract
A joint project between the Canadian Astronomy Data Center of the National Research Council Canada, and the italian Istituto Nazionale di Astrofisica-Osservatorio Astronomico di Trieste (INAF-OATs), partially funded by the EGI-Engage H2020 European Project, is devoted to deploy an integrated infrastructure, based on the International Virtual Observatory Alliance (IVOA) standards, to access and exploit astronomical data. Currently CADC-CANFAR provides scientists with an access, storage and computation facility, based on software libraries implementing a set of standards developed by the International Virtual Observatory Alliance (IVOA). The deployment of a twin infrastructure, basically built on the same open source software libraries, has been started at INAF-OATs. This new infrastructure now provides users with an Access Control Service and a Storage Service. The final goal of the ongoing project is to build an integrated infrastructure geographycally distributed providing complete interoperability, both in users access control and data sharing. This paper describes the target infrastructure, the main user requirements covered, the technical choices and the implemented solutions.
rate research
Read More
Handling, processing and archiving the huge amount of data produced by the new generation of experiments and instruments in Astronomy and Astrophysics are among the more exciting challenges to address in designing the future data management infrastructures and computing services. We investigated the feasibility of a data management and computation infrastructure, available world-wide, with the aim of merging the FAIR (Findable, Accessible, Interoperable, Reusable) data management provided by IVOA standards with the efficiency and reliability of a cloud approach. Our work involved the Canadian Advanced Network for Astronomy Research (CANFAR) infrastructure and the European EGI federated cloud (EFC). We designed and deployed a pilot data management and computation infrastructure that provides IVOA-compliant VOSpace storage resources and wide access to interoperable federated clouds. In this paper, we detail the main user requirements covered, the technical choices and the implemented solutions and we describe the resulting Hybrid cloud Worldwide infrastructure, its benefits and limitations.
The increase of astronomical data produced by a new generation of observational tools poses the need to distribute data and to bring computation close to the data. Trying to answer this need, we set up a federated data and computing infrastructure involving an international cloud facility, EGI federated, and a set of services implementing IVOA standards and recommendations for authentication, data sharing and resource access. In this paper we describe technical problems faced, specifically we show the designing, technological and architectural solutions adopted. We depict our technological overall solution to bring data close to computation resources. Besides the adopted solutions, we propose some points for an open discussion on authentication and authorization mechanisms.
In a distributed storage system, code symbols are dispersed across space in nodes or storage units as opposed to time. In settings such as that of a large data center, an important consideration is the efficient repair of a failed node. Efficient repair calls for erasure codes that in the face of node failure, are efficient in terms of minimizing the amount of repair data transferred over the network, the amount of data accessed at a helper node as well as the number of helper nodes contacted. Coding theory has evolved to handle these challenges by introducing two new classes of erasure codes, namely regenerating codes and locally recoverable codes as well as by coming up with novel ways to repair the ubiquitous Reed-Solomon code. This survey provides an overview of the efforts in this direction that have taken place over the past decade.
Recently, the research on local repair codes is mainly confined to repair the failed nodes within each repair group. But if the extreme cases occur that the entire repair group has failed, the local code stored in the failed group need to be recovered as a whole. In this paper, local codes with cooperative repair, in which the local codes are constructed based on minimum storage regeneration (MSR) codes, is proposed to achieve repairing the failed groups. Specifically, the proposed local codes with cooperative repair construct a kind of mutual interleaving structure among the parity symbols, that the parity symbols of each local code, named as distributed local parity, can be generated by the parity symbols of the MSR codes in its two adjacent local codes. Taking advantage of the structure given, the failed local groups can be repaired cooperatively by their adjacent local groups with lower repair locality, and meanwhile the minimum distance of local codes with cooperative repair is derived. Theoretical analysis and simulation experiments show that, compared with codes with local regeneration (such as MSR-local codes and MBR-local codes), the proposed local codes with cooperative repair have benefits in bandwidth overhead and repair locality for the case of local groups failure.
This paper presents an approach for constructing distributed storage system based on micro-service architecture. By building storage functionalities using micro services, we can provide new capabilities to a distributed storage system in a flexible way. We take erasure coding and compression as two case studies to show how to build a micro-service based distributed storage system. We also show that by building erasure coding and compression as micro-services, the distributed storage system still achieves reasonable performance compared to the monolithic one.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
