Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLearning How to Ask: Querying LMs with Mixtures of Soft Prompts
تعلم كيفية السؤال: الاستعلام LMS مع مخاليط من المطالبات الناعمة
709
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تم استخدام مطالبات اللغة الطبيعية مؤخرا لتخصيص نماذج اللغة في أداء مهام منظمة العفو الدولية الأخرى، باستخدام نموذج تعبئة داخل الفراغ (Petroni et al.، 2019) أو نموذج استقراء قليل بالرصاص (براون وآخرون، 2020). على سبيل المثال، تحتفظ نماذج اللغة بالمعرفة الواقعية من كورسا التدريب التي يمكن استخراجها من خلال مطالبتها بملء الفراغ "في موجه أساسية. ومع ذلك، أين يأتي هذا المطالبة؟ نستكشف فكرة مطالبات التعلم عن طريق نزول التدرج --- إما مطالبات ضبط دقيقة مأخوذة من العمل السابق، أو بدءا من تهيئة عشوائية. تتكون مطالباتنا من كلمات ناعمة، '' I.E.، ناقلات مستمرة ليست بالضرورة تضمين نوع الكلمات من نموذج اللغة. علاوة على ذلك، لكل مهمة، فإننا نحسن مزيجا من المطالبات، والتعلم الذي يطالب الأكثر فعالية وكيفية الفرقة لهم. عبر العديد من LMS والمهام الإنجليزية المتعددة، يتفوق نهجنا بشكل كبير على الأساليب السابقة، مما يظهر أن المعرفة الواقعية الضمنية في نماذج اللغة قد تم التقليل من السابق. علاوة على ذلك، فإن هذه المعرفة رخيصة للاستيلاء: تهيئة عشوائية جيدة مثل التهيئة المستنيرة.
Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks, using a fill-in-the-blank paradigm (Petroni et al., 2019) or a few-shot extrapolation paradigm (Brown et al., 2020). For example, language models retain factual knowledge from their training corpora that can be extracted by asking them to fill in the blank'' in a sentential prompt. However, where does this prompt come from? We explore the idea of learning prompts by gradient descent---either fine-tuning prompts taken from previous work, or starting from random initialization. Our prompts consist of soft words,'' i.e., continuous vectors that are not necessarily word type embeddings from the language model. Furthermore, for each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them. Across multiple English LMs and tasks, our approach hugely outperforms previous methods, showing that the implicit factual knowledge in language models was previously underestimated. Moreover, this knowledge is cheap to elicit: random initialization is nearly as good as informed initialization.
References used
https://aclanthology.org/
rate research
Read More
We present a scaffolded discovery learning approach to introducing concepts in a Natural Language Processing course aimed at computer science students at liberal arts institutions. We describe some of the objectives of this approach, as well as prese
nting specific ways that four of our discovery-based assignments combine specific natural language processing concepts with broader analytic skills. We argue this approach helps prepare students for many possible future paths involving both application and innovation of NLP technology by emphasizing experimental data navigation, experiment design, and awareness of the complexities and challenges of analysis.
Myelomeningoceles are very common anamoly in our country. Mostly it ends
with permanent damage and handicap. Lot of these children die due to meningitis as a complication.
It still till now a large number of children with myelo meningoceles seek me
dical care in pediatric hospital
and other health centers. So, we must know the reasons and the predisposing factors for the
myelomeningoceles to reduce their incidence.
Abstract We introduce an Edit-Based TransfOrmer with Repositioning (EDITOR), which makes sequence generation flexible by seamlessly allowing users to specify preferences in output lexical choice. Building on recent models for non-autoregressive seque
nce generation (Gu et al., 2019), EDITOR generates new sequences by iteratively editing hypotheses. It relies on a novel reposition operation designed to disentangle lexical choice from word positioning decisions, while enabling efficient oracles for imitation learning and parallel edits at decoding time. Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer (Gu et al., 2019) while speeding up decoding dramatically compared to constrained beam search (Post and Vilar, 2018). EDITOR also achieves comparable or better translation quality with faster decoding speed than the Levenshtein Transformer on standard Romanian-English, English-German, and English-Japanese machine translation tasks.
An important task in NLP applications such as sentence simplification is the ability to take a long, complex sentence and split it into shorter sentences, rephrasing as necessary. We introduce a novel dataset and a new model for this split and rephra
se' task. Our BiSECT training data consists of 1 million long English sentences paired with shorter, meaning-equivalent English sentences. We obtain these by extracting 1-2 sentence alignments in bilingual parallel corpora and then using machine translation to convert both sides of the corpus into the same language. BiSECT contains higher quality training examples than the previous Split and Rephrase corpora, with sentence splits that require more significant modifications. We categorize examples in our corpus and use these categories in a novel model that allows us to target specific regions of the input sentence to be split and edited. Moreover, we show that models trained on BiSECT can perform a wider variety of split operations and improve upon previous state-of-the-art approaches in automatic and human evaluations.
While pre-trained language models (PLMs) are the go-to solution to tackle many natural language processing problems, they are still very limited in their ability to capture and to use common-sense knowledge. In fact, even if information is available
in the form of approximate (soft) logical rules, it is not clear how to transfer it to a PLM in order to improve its performance for deductive reasoning tasks. Here, we aim to bridge this gap by teaching PLMs how to reason with soft Horn rules. We introduce a classification task where, given facts and soft rules, the PLM should return a prediction with a probability for a given hypothesis. We release the first dataset for this task, and we propose a revised loss function that enables the PLM to learn how to predict precise probabilities for the task. Our evaluation results show that the resulting fine-tuned models achieve very high performance, even on logical rules that were unseen at training. Moreover, we demonstrate that logical notions expressed by the rules are transferred to the fine-tuned model, yielding state-of-the-art results on external datasets.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
