Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userEditing Factual Knowledge in Language Models
تحرير المعرفة الواقعية في نماذج اللغة
508
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
المعرفة الواقعية المكتسبة أثناء التدريب المسبق وتخزينها في معلمات نماذج اللغة (LMS) يمكن أن تكون مفيدة في مهام المصب (على سبيل المثال، الإجابة على السؤال أو الاستدلال النصي). ومع ذلك، يمكن أن تسبب بعض الحقائق أو تصبح عفا عليها الزمن مع مرور الوقت. نقدم المعلمين، وهي طريقة يمكن استخدامها لتحرير هذه المعرفة، وبالتالي إصلاح الأخطاء أو التنبؤات غير المتوقعة دون الحاجة إلى إعادة تدريب مكلفة أو ضبط جيد. إلى جانب كونها فعالة بشكل حسابي، لا تتطلب المعرفة المعرفة أي تعديلات في LM قبل التدريب (على سبيل المثال، استخدام التعلم التلوي). في نهجنا، نحن ندرب شبكة فرط بتحسين مقيد لتعديل حقيقة دون التأثير على بقية المعرفة؛ ثم يتم استخدام شبكة Hyper المدربة للتنبؤ بتحديث الوزن في وقت الاختبار. نعرض فعالية المعرفة مع اثنين من المهندسينيات الشائعة ومهام المعرفة المكثفة: ط) نموذج بيرت يتم ضبطه بشكل جيد لفحص الحقائق، والثاني) نموذج بارت تسلسل إلى تسلسل للرد على السؤال. من خلال طريقتنا، يميل تغيير التنبؤ بشأن الصياغة المحددة لاستعلامه إلى تغيير متسق في التنبؤ أيضا بصيادتها. نظرا لأن هذا يمكن تشجيعه بشكل أكبر من خلال استغلال الصياغة (على سبيل المثال، التي تم إنشاؤها تلقائيا) أثناء التدريب. ومن المثير للاهتمام، أن شبكة فرط لدينا يمكن اعتبارها مسبار "تكشف عن مكونات يجب تغييرها لمعالجة المعرفة الواقعية؛ يوضح تحليلنا أن التحديثات تميل إلى التركيز على مجموعة فرعية صغيرة من المكونات. شفرة المصدر المتاحة في https://github.com/nicola-decao/knowegleditor
The factual knowledge acquired during pre-training and stored in the parameters of Language Models (LMs) can be useful in downstream tasks (e.g., question answering or textual inference). However, some facts can be incorrectly induced or become obsolete over time. We present KnowledgeEditor, a method which can be used to edit this knowledge and, thus, fix bugs' or unexpected predictions without the need for expensive re-training or fine-tuning. Besides being computationally efficient, KnowledgeEditordoes not require any modifications in LM pre-training (e.g., the use of meta-learning). In our approach, we train a hyper-network with constrained optimization to modify a fact without affecting the rest of the knowledge; the trained hyper-network is then used to predict the weight update at test time. We show KnowledgeEditor's efficacy with two popular architectures and knowledge-intensive tasks: i) a BERT model fine-tuned for fact-checking, and ii) a sequence-to-sequence BART model for question answering. With our method, changing a prediction on the specific wording of a query tends to result in a consistent change in predictions also for its paraphrases. We show that this can be further encouraged by exploiting (e.g., automatically-generated) paraphrases during training. Interestingly, our hyper-network can be regarded as a probe' revealing which components need to be changed to manipulate factual knowledge; our analysis shows that the updates tend to be concentrated on a small subset of components. Source code available at https://github.com/nicola-decao/KnowledgeEditor
References used
https://aclanthology.org/
rate research
Read More
Relational knowledge bases (KBs) are commonly used to represent world knowledge in machines. However, while advantageous for their high degree of precision and interpretability, KBs are usually organized according to manually-defined schemas, which l
imit their expressiveness and require significant human efforts to engineer and maintain. In this review, we take a natural language processing perspective to these limitations, examining how they may be addressed in part by training deep contextual language models (LMs) to internalize and express relational knowledge in more flexible forms. We propose to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision. Our contributions are threefold: (1) We provide a high-level, extensible taxonomy for knowledge representation in LMs; (2) Within our taxonomy, we highlight notable models, evaluation tasks, and findings, in order to provide an up-to-date review of current knowledge representation capabilities in LMs; and (3) We suggest future research directions that build upon the complementary aspects of LMs and KBs as knowledge representations.
Factual inconsistencies existed in the output of abstractive summarization models with original documents are frequently presented. Fact consistency assessment requires the reasoning capability to find subtle clues to identify whether a model-generat
ed summary is consistent with the original document. This paper proposes a fine-grained two-stage Fact Consistency assessment framework for Summarization models (SumFC). Given a document and a summary sentence, in the first stage, SumFC selects the top-K most relevant sentences with the summary sentence from the document. In the second stage, the model performs fine-grained consistency reasoning at the sentence level, and then aggregates all sentences' consistency scores to obtain the final assessment result. We get the training data pairs by data synthesis and adopt contrastive loss of data pairs to help the model identify subtle cues. Experiment results show that SumFC has made a significant improvement over the previous state-of-the-art methods. Our experiments also indicate that SumFC distinguishes detailed differences better.
The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large
KGs, and (ii) perform joint reasoning over the QA context and KG. Here we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph-based message passing. We evaluate QA-GNN on the CommonsenseQA and OpenBookQA datasets, and show its improvement over existing LM and LM+KG models, as well as its capability to perform interpretable and structured reasoning, e.g., correctly handling negation in questions.
Prior work has shown that structural supervision helps English language models learn generalizations about syntactic phenomena such as subject-verb agreement. However, it remains unclear if such an inductive bias would also improve language models' a
bility to learn grammatical dependencies in typologically different languages. Here we investigate this question in Mandarin Chinese, which has a logographic, largely syllable-based writing system; different word order; and sparser morphology than English. We train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and Transformer-parameterized generative parsing models on two Mandarin Chinese datasets of different sizes. We evaluate the models' ability to learn different aspects of Mandarin grammar that assess syntactic and semantic relationships. We find suggestive evidence that structural supervision helps with representing syntactic state across intervening content and improves performance in low-data settings, suggesting that the benefits of hierarchical inductive biases in acquiring dependency relationships may extend beyond English.
The key challenge of question answering over knowledge bases (KBQA) is the inconsistency between the natural language questions and the reasoning paths in the knowledge base (KB). Recent graph-based KBQA methods are good at grasping the topological s
tructure of the graph but often ignore the textual information carried by the nodes and edges. Meanwhile, pre-trained language models learn massive open-world knowledge from the large corpus, but it is in the natural language form and not structured. To bridge the gap between the natural language and the structured KB, we propose three relation learning tasks for BERT-based KBQA, including relation extraction, relation matching, and relation reasoning. By relation-augmented training, the model learns to align the natural language expressions to the relations in the KB as well as reason over the missing connections in the KB. Experiments on WebQSP show that our method consistently outperforms other baselines, especially when the KB is incomplete.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
