Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userHow Can We Know What Language Models Know?
205
0
0.0
(
0
)
Added by
Zhengbao Jiang
Publication date
2019
fields
Informatics Engineering
and research's language is
English
Ask ChatGPT about the research
No Arabic abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LM) by having the LM fill in the blanks of prompts such as Obama is a _ by profession. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as Obama worked as a _ may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA.
rate research
Read More
A neural network deployed in the wild may be asked to make predictions for inputs that were drawn from a different distribution than that of the training data. A plethora of work has demonstrated that it is easy to find or synthesize inputs for which a neural network is highly confident yet wrong. Generative models are widely viewed to be robust to such mistaken confidence as modeling the density of the input features can be used to detect novel, out-of-distribution inputs. In this paper we challenge this assumption. We find that the density learned by flow-based models, VAEs, and PixelCNNs cannot distinguish images of common objects such as dogs, trucks, and horses (i.e. CIFAR-10) from those of house numbers (i.e. SVHN), assigning a higher likelihood to the latter when the model is trained on the former. Moreover, we find evidence of this phenomenon when pairing several popular image data sets: FashionMNIST vs MNIST, CelebA vs SVHN, ImageNet vs CIFAR-10 / CIFAR-100 / SVHN. To investigate this curious behavior, we focus analysis on flow-based generative models in particular since they are trained and evaluated via the exact marginal likelihood. We find such behavior persists even when we restrict the flows to constant-volume transformations. These transformations admit some theoretical analysis, and we show that the difference in likelihoods can be explained by the location and variances of the data and the model curvature. Our results caution against using the density estimates from deep generative models to identify inputs similar to the training distribution until their behavior for out-of-distribution inputs is better understood.
String theory has transformed our understanding of geometry, topology and spacetime. Thus, for this special issue of Foundations of Physics commemorating Forty Years of String Theory, it seems appropriate to step back and ask what we do not understand. As I will discuss, time remains the least understood concept in physical theory. While we have made significant progress in understanding space, our understanding of time has not progressed much beyond the level of a century ago when Einstein introduced the idea of space-time as a combined entity. Thus, I will raise a series of open questions about time, and will review some of the progress that has been made as a roadmap for the future.
In the present paper, we investigate the cosmographic problem using the bias-variance trade-off. We find that both the z-redshift and the $y=z/(1+z)$-redshift can present a small bias estimation. It means that the cosmography can describe the supernova data more accurately. Minimizing risk, it suggests that cosmography up to the second order is the best approximation. Forecasting the constraint from future measurements, we find that future supernova and redshift drift can significantly improve the constraint, thus having the potential to solve the cosmographic problem. We also exploit the values of cosmography on the deceleration parameter and equation of state of dark energy $w(z)$. We find that supernova cosmography cannot give stable estimations on them. However, much useful information was obtained, such as that the cosmography favors a complicated dark energy with varying $w(z)$, and the derivative $dw/dz<0$ for low redshift. The cosmography is helpful to model the dark energy.
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuAD 2.0, the latest version of the Stanford Question Answering Dataset (SQuAD). SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuAD 2.0 is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD 1.1 achieves only 66% F1 on SQuAD 2.0.
We investigate possibility of emission of the bremsstrahlung photons in nuclear reactions with hypernuclei for the first time. A new model of the bremsstrahlung emission which accompanies interactions between $alpha$ particles and hypernuclei is constructed, where a new formalism for the magnetic momenta of nucleons and hyperon inside hypernucleus is added. For first calculations, we choose $alpha$ decay of the normal nucleus $^{210}{rm Po}$ and the hypernucleus $^{211}_{Lambda}{rm Po}$. We find that (1) emission for the hypernucleus $^{211}_{Lambda}{rm Po}$ is larger than for normal nucleus $^{210}{rm Po}$, (2) difference between these spectra is small. We propose a way how to find hypernuclei, where role of hyperon is the most essential in emission of bremsstrahlung photons during $alpha$ decay. As demonstration of such a property, we show that the spectra for the hypernuclei $^{107}_{Lambda}{rm Te}$ and $^{109}_{Lambda}{rm Te}$ are essentially larger than the spectra for the normal nuclei $^{106}{rm Te}$ and $^{108}{rm Te}$. Such a difference is explained by additional contribution of emission to the full bremsstrahlung, which is formed by magnetic moment of hyperon inside hypernucleus. The bremsstrahlung emission formed by such a mechanism, is of the magnetic type. A new formula for fast estimations of bremsstrahlung spectra for even-even hypernuclei is proposed, where role of magnetic moment of hyperon of hypernucleus in formation of the bremsstrahlung emission is shown explicitly. Such an analysis opens possibility of new experimental study of properties of hypernuclei via bremsstrahlung study.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
