We explore the utility of Karhunen Loeve (KL) analysis in solving practical problems in the analysis of gravitational shear surveys. Shear catalogs from large-field weak lensing surveys will be subject to many systematic limitations, notably incomplete coverage and pixel-level masking due to foreground sources. We develop a method to use two dimensional KL eigenmodes of shear to interpolate noisy shear measurements across masked regions. We explore the results of this method with simulated shear catalogs, using statistics of high-convergence regions in the resulting map. We find that the KL procedure not only minimizes the bias due to masked regions in the field, it also reduces spurious peak counts from shape noise by a factor of ~ 3 in the cosmologically sensitive regime. This indicates that KL reconstructions of masked shear are not only useful for creating robust convergence maps from masked shear catalogs, but also offer promise of improved parameter constraints within studies of shear peak statistics.
Supernova rates are directly coupled to high mass stellar birth and evolution. As such, they are one of the few direct measures of the history of cosmic stellar evolution. In this paper we describe an probabilistic technique for identifying supernovae within spectroscopic samples of galaxies. We present a study of 52 type Ia supernovae ranging in age from -14 days to +40 days extracted from a parent sample of simeq 50,000 spectra from the SDSS DR5. We find a Supernova Rate (SNR) of 0.472^{+0.048}_{-0.039}(Systematic)^{+0.081}_{-0.071}(Statistical)SNu at a redshift of <z> = 0.1. This value is higher than other values at low redshift at the 1{sigma}, but is consistent at the 3{sigma} level. The 52 supernova candidates used in this study comprise the third largest sample of supernovae used in a type Ia rate determination to date. In this paper we demonstrate the potential for the described approach for detecting supernovae in future spectroscopic surveys.
In the coming decade, astronomical surveys of the sky will generate tens of terabytes of images and detect hundreds of millions of sources every night. The study of these sources will involve computation challenges such as anomaly detection and classification, and moving object tracking. Since such studies benefit from the highest quality data, methods such as image coaddition (stacking) will be a critical preprocessing step prior to scientific investigation. With a requirement that these images be analyzed on a nightly basis to identify moving sources or transient objects, these data streams present many computational challenges. Given the quantity of data involved, the computational load of these problems can only be addressed by distributing the workload over a large number of nodes. However, the high data throughput demanded by these applications may present scalability challenges for certain storage architectures. One scalable data-processing method that has emerged in recent years is MapReduce, and in this paper we focus on its popular open-source implementation called Hadoop. In the Hadoop framework, the data is partitioned among storage attached directly to worker nodes, and the processing workload is scheduled in parallel on the nodes that contain the required input data. A further motivation for using Hadoop is that it allows us to exploit cloud computing resources, e.g., Amazons EC2. We report on our experience implementing a scalable image-processing pipeline for the SDSS imaging database using Hadoop. This multi-terabyte imaging dataset provides a good testbed for algorithm development since its scope and structure approximate future surveys. First, we describe MapReduce and how we adapted image coaddition to the MapReduce framework. Then we describe a number of optimizations to our basic approach and report experimental results comparing their performance.
We present a new method for constructing three-dimensional mass maps from gravitational lensing shear data. We solve the lensing inversion problem using truncation of singular values (within the context of generalized least squares estimation) without a priori assumptions about the statistical nature of the signal. This singular value framework allows a quantitative comparison between different filtering methods: we evaluate our method beside the previously explored Wiener filter approaches. Our method yields near-optimal angular resolution of the lensing reconstruction and allows cluster sized halos to be de-blended robustly. It allows for mass reconstructions which are 2-3 orders-of-magnitude faster than the Wiener filter approach; in particular, we estimate that an all-sky reconstruction with arcminute resolution could be performed on a time-scale of hours. We find however that linear, non-parametric reconstructions have a fundamental limitation in the resolution achieved in the redshift direction.
Under the unified model for active galactic nuclei (AGNs), narrow-line (Type 2) AGNs are, in fact, broad-line (Type 1) AGNs but each with a heavily obscured accretion disk. We would therefore expect the optical continuum emission from Type 2 AGN to be composed mainly of stellar light and non-variable on the time-scales of months to years. In this work we probe the spectroscopic variability of galaxies and narrow-line AGNs using the multi-epoch data in the Sloan Digital Sky Survey (SDSS) Data Release 6. The sample contains 18,435 sources for which there exist pairs of spectroscopic observations (with a maximum separation in time of ~700 days) covering a wavelength range of 3900-8900 angstrom. To obtain a reliable repeatability measurement between each spectral pair, we consider a number of techniques for spectrophotometric calibration resulting in an improved spectrophotometric calibration of a factor of two. From these data we find no obvious continuum and emission-line variability in the narrow-line AGNs on average -- the spectroscopic variability of the continuum is 0.07+/-0.26 mag in the g band and, for the emission-line ratios log10([NII]/Halpha) and log10([OIII]/Hbeta), the variability is 0.02+/-0.03 dex and 0.06+/-0.08 dex, respectively. From the continuum variability measurement we set an upper limit on the ratio between the flux of varying spectral component, presumably related to AGN activities, and that of host galaxy to be ~30%. We provide the corresponding upper limits for other spectral classes, including those from the BPT diagram, eClass galaxy classification, stars and quasars.
We use galaxy surface brightness as prior information to improve photometric redshift (photo-z) estimation. We apply our template-based photo-z method to imaging data from the ground-based VVDS survey and the space-based GOODS field from HST, and use spectroscopic redshifts to test our photometric redshifts for different galaxy types and redshifts. We find that the surface brightness prior eliminates a large fraction of outliers by lifting the degeneracy between the Lyman and 4000 Angstrom breaks. Bias and scatter are improved by about a factor of 2 with the prior for both the ground and space data. Ongoing and planned surveys from the ground and space will benefit, provided that care is taken in measurements of galaxy sizes and in the application of the prior. We discuss the image quality and signal-to-noise requirements that enable the surface brightness prior to be successfully applied.
Virtual observatories will give astronomers easy access to an unprecedented amount of data. Extracting scientific knowledge from these data will increasingly demand both efficient algorithms as well as the power of parallel computers. Nearly all efficient analyses of large astronomical datasets use trees as their fundamental data structure. Writing efficient tree-based techniques, a task that is time-consuming even on single-processor computers, is exceedingly cumbersome on massively parallel platforms (MPPs). Most applications that run on MPPs are simulation codes, since the expense of developing them is offset by the fact that they will be used for many years by many researchers. In contrast, data analysis codes change far more rapidly, are often unique to individual researchers, and therefore accommodate little reuse. Consequently, the economics of the current high-performance computing development paradigm for MPPs does not favor data analysis applications. We have therefore built a library, called Ntropy, that provides a flexible, extensible, and easy-to-use way of developing tree-based data analysis algorithms for both serial and parallel platforms. Our experience has shown that not only does our library save development time, it can also deliver excellent serial performance and parallel scalability. Furthermore, Ntropy makes it easy for an astronomer with little or no parallel programming experience to quickly scale their application to a distributed multiprocessor environment. By minimizing development time for efficient and scalable data analysis, we enable wide-scale knowledge discovery on massive datasets.
Astronomy began as a visual science, first through careful observations of the sky using either an eyepiece or the naked eye, then on to the preservation of those images with photographic media and finally the digital encoding of that information via CCDs. This last step has enabled astronomy to move into a fully automated era -- where data is recorded, analyzed and interpreted often without any direct visual inspection. Sky in Google Earth completes that circle by providing an intuitive visual interface to some of the largest astronomical imaging surveys covering the full sky. By streaming imagery, catalogs, time domain data, and ancillary information directly to a user, Sky can provide the general public as well as professional and amateur astronomers alike with a wealth of information for use in education and research. We provide here a brief introduction to Sky in Google Earth, focusing on its extensible environment, how it may be integrated into the research process and how it can bring astronomical research to a broader community. With an open interface available on Linux, Mac OS X and Windows, applications developed within Sky are accessible not just within the Google framework but through any visual browser that supports the Keyhole Markup Language. We present Sky as the embodiment of a virtual telescope.
