اشترك بالحزمة الذهبية واحصل على وصول غير محدود شمرا أكاديميا
تسجيل مستخدم جديدEnhanced Remote Astronomical Archive System Based on the File-Level Unlimited Sliding-Window Technique
57
0
0.0
(
0
)
اسأل ChatGPT حول البحث
ﻻ يوجد ملخص باللغة العربية
Data archiving is one of the most critical issues for modern astronomical observations. With the development of a new generation of radio telescopes, the transfer and archiving of massive remote data have become urgent problems to be solved. Herein, we present a practical and robust file-level flow-control approach, called the Unlimited Sliding-Window (USW), by referring to the classic flow-control method in TCP protocol. Basing on the USW and the Next Generation Archive System (NGAS) developed for the Murchison Widefield Array telescope, we further implemented an enhanced archive system (ENGAS) using ZeroMQ middleware. The ENGAS substantially improves the transfer performance and ensures the integrity of transferred files. In the tests, the ENGAS is approximately three to twelve times faster than the NGAS and can fully utilize the bandwidth of network links. Thus, for archiving radio observation data, the ENGAS reduces the communication time, improves the bandwidth utilization, and solves the remote synchronous archiving of data from observatories such as Mingantu spectral radioheliograph. It also provides a better reference for the future construction of the Square Kilometer Array (SKA) Science Regional Center.
قيم البحث
اقرأ أيضاً
The FITS is the standard file format in astronomy, and it has been extended to agree with astronomical needs of the day. However, astronomical datasets have been inflating year by year. In case of ALMA telescope, a ~ TB scale 4-dimensional data cube
may be produced for one target. Considering that typical Internet bandwidth is a few 10 MB/s at most, the original data cubes in FITS format are hosted on a VO server, and the region which a user is interested in should be cut out and transferred to the user (Eguchi et al., 2012). The system will equip a very high-speed disk array to process a TB scale data cube in a few 10 seconds, and disk I/O speed, endian conversion and data processing one will be comparable. Hence to reduce the endian conversion time is one of issues to realize our system. In this paper, I introduce a technique named just-in-time endian conversion, which delays the endian conversion for each pixel just before it is really needed, to sweep out the endian conversion time; by applying this method, the FITS processing speed increases 20% for single threading, and 40% for multi-threading compared to CFITSIO. The speed-up by the method tightly relates to modern CPU architecture to improve the efficiency of instruction pipelines due to break of causality, a programmed instruction code sequence.
Dome A, Antarctica has been thought to be one of the best astronomical sites on the earth since decades ago. From it was first visited by astronomers in 2008, dozens of facilities for astronomical observation and site testing were deployed. Due to it
s special geographical location, the data and message exchange between Dome A and the domestic control center could only depend on Iridium. Because the link bandwidth of Iridium is extremely limited, meanwhile the network traffic cost is quite expensive and the network is rather unstable, the commonly used data transfer tools, such as rsync and scp, are not suitable in this case. In this paper, we design and implement a data transfer tool called NBFTP (narrow bandwidth file transfer protocol) for the astronomical observation of Dome A. NBFTP uses a uniform interface to arrange all types of data and matches specific transmission schemes for different data types according to rules. Break-point resuming and extensibility functions are also implemented. Our experimental results show that NBFTP consumes 60% less network traffic than rsync when detecting the data pending to be transferred. And when transferring small files of 1KB, the network traffic consumption of NBFTP is 40% less than rsync. However, as the file size increases, the network traffic consumption of NBFTP tends to approach rsync, but it is still smaller than rsync.
The world astronomical image archives represent huge opportunities to time-domain astronomy sciences and other hot topics such as space defense, and astronomical observatories should improve this wealth and make it more accessible in the big data era
. In 2010 we introduced the Mega-Archive database and the Mega-Precovery server for data mining images containing Solar system bodies, with focus on near Earth asteroids (NEAs). This paper presents the improvements and introduces some new related data mining tools developed during the last five years. Currently, the Mega-Archive has indexed 15 million images available from six major collections (CADC, ESO, ING, LCOGT, NVO and SMOKA) and other instrument archives and surveys. This meta-data index collection is daily updated (since 2014) by a crawler which performs automated query of five major collections. Since 2016, these data mining tools run to the new dedicated EURONEAR server, and the database migrated to SQL engine which supports robust and fast queries. To constrain the area to search moving or fixed objects in images taken by large mosaic cameras, we built the graphical tools FindCCD and FindCCD for Fixed Objects which overlay the targets across one of seven mosaic cameras (Subaru-SuprimeCam, VST-OmegaCam, INT-WFC, VISTA-VIRCAM, CFHT-MegaCam, Blanco-DECam and Subaru-HSC), also plotting the uncertainty ellipse for poorly observed NEAs. In 2017 we improved Mega-Precovery, which offers now two options for calculus of the ephemerides and three options for the input (objects defined by designation, orbit or observations). Additionally, we developed Mega-Archive for Fixed Objects (MASFO) and Mega-Archive Search for Double Stars (MASDS). We believe that the huge potential of science imaging archives is still insufficiently exploited.
Astronomical observation data require long-term preservation, and the rapid accumulation of observation data makes it necessary to consider the cost of long-term archive storage. In addition to low-speed disk-based online storage, optical disk or tap
e-based offline storage can be used to save costs. However, for astronomical research that requires historical data (particularly time-domain astronomy), the performance and energy consumption of data-accessing techniques cause problems because the requested data (which are organized according to observation time) may be located across multiple storage devices. In this study, we design and develop a tool referred to as AstroLayout to redistribute the observation data using spatial aggregation. The core algorithm uses graph partitioning to generate an optimized data placement according to the original observation data statistics and the target storage system. For the given observation data, AstroLayout can copy the long-term archive in the target storage system in accordance with this placement. An efficiency evaluation shows that AstroLayout can reduce the number of devices activated when responding to data-access requests in time-domain astronomy research. In addition to improving the performance of data-accessing techniques, AstroLayout can also reduce the storage systems power consumption. For enhanced adaptability, it supports storage systems of any media, including optical disks, tapes, and hard disks.
The experimental issue of the search for new particles of unknown mass poses the challenge of exploring a wide interval to look for the usual signatures represented by excess of events above the background. A side effect of such a broad range quest i
s that the traditional significance calculations valid for signals of known location are no more applicable when such an information is missing. In this note the specific signal search approach via observation windows sliding over the range of interest is considered; in the assumptions of known background and of fixed width of the exploring windows the statistical implications of such a search scheme are described, with special emphasis on the correct significance assessment for a claimed discovery.
سجل دخول لتتمكن من نشر تعليقات
التعليقات
جاري جلب التعليقات
سجل دخول لتتمكن من متابعة معايير البحث التي قمت باختيارها
