Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userSimultaneously Self-Attending to Text and Entities for Knowledge-Informed Text Representations
في وقت واحد حضور الذات للنص والكيانات للتمثيلات النصية المعرفة
49
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
برزت نماذج اللغة المدربة مسبقا كطرق ناجحة للغاية لتعلم تمثيلات نصية جيدة. ومع ذلك، فإن كمية المعرفة المنظمة المحتجزة في هذه النماذج، وكيف يمكن استخراج (إذا كان على الإطلاق)، فلا يزال سؤالا مفتوحا. في هذا العمل، نهدف إلى التعلم مباشرة من التمثيلات النصية التي ترفض المعرفة المنظمة حول الكيانات المذكورة في النص. هذا يمكن أن يكون مفيدا بشكل خاص لمهام المصب التي هي مكثفة المعرفة. يستخدم نهجنا اهتماما الذات بين الكلمات في كيانات الرسم البياني للنص والمعرفة (KG) المذكورة في النص. في حين تتطلب الأساليب الحالية بيانات مرتبطة بالكياء للتدريب المسبق، فإننا نتدرب باستخدام هدف إخفاء الإشارة وهدف المرشح - الذي لا يتطلب أي روابط كيانا وفتح فقط الوصول إلى جدول الاسم المستعار لاسترداد المرشحين، تمكين ما قبل التدريب على نطاق واسع. نظهر أن النموذج المقترح يتعلم تمثيلات النص المستنيرة بالمعرفة التي تسفر عن تحسينات على مهام المصب على الأساليب الحالية.
Pre-trained language models have emerged as highly successful methods for learning good text representations. However, the amount of structured knowledge retained in such models, and how (if at all) it can be extracted, remains an open question. In this work, we aim at directly learning text representations which leverage structured knowledge about entities mentioned in the text. This can be particularly beneficial for downstream tasks which are knowledge-intensive. Our approach utilizes self-attention between words in the text and knowledge graph (KG) entities mentioned in the text. While existing methods require entity-linked data for pre-training, we train using a mention-span masking objective and a candidate ranking objective -- which doesn't require any entity-links and only assumes access to an alias table for retrieving candidates, enabling large-scale pre-training. We show that the proposed model learns knowledge-informed text representations that yield improvements on the downstream tasks over existing methods.
References used
https://aclanthology.org/
rate research
Read More
This paper describes NAIST's system for the English-to-Japanese Simultaneous Text-to-text Translation Task in IWSLT 2021 Evaluation Campaign. Our primary submission is based on wait-k neural machine translation with sequence-level knowledge distillation to encourage literal translation.
In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summa
rization model is trained with guidance from a teacher which generates smoothed labels to help regularize training. Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.
Learning to capture text-table alignment is essential for tasks like text-to-SQL. A model needs to correctly recognize natural language references to columns and values and to ground them in the given database schema. In this paper, we present a nove
l weakly supervised Structure-Grounded pretraining framework (STRUG) for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-table corpus. We identify a set of novel pretraining tasks: column grounding, value grounding and column-value mapping, and leverage them to pretrain a text-table encoder. Additionally, to evaluate different methods under more realistic text-table alignment settings, we create a new evaluation set Spider-Realistic based on Spider dev set with explicit mentions of column names removed, and adopt eight existing text-to-SQL datasets for cross-database evaluation. STRUG brings significant improvement over BERTLARGE in all settings. Compared with existing pretraining methods such as GRAPPA, STRUG achieves similar performance on Spider, and outperforms all baselines on more realistic sets. All the code and data used in this work will be open-sourced to facilitate future research.
Deep-learning models for language generation tasks tend to produce repetitive output. Various methods have been proposed to encourage lexical diversity during decoding, but this often comes at a cost to the perceived fluency and adequacy of the outpu
t. In this work, we propose to ameliorate this cost by using an Imitation Learning approach to explore the level of diversity that a language generation model can reliably produce. Specifically, we augment the decoding process with a meta-classifier trained to distinguish which words at any given timestep will lead to high-quality output. We focus our experiments on concept-to-text generation where models are sensitive to the inclusion of irrelevant words due to the strict relation between input and output. Our analysis shows that previous methods for diversity underperform in this setting, while human evaluation suggests that our proposed method achieves a high level of diversity with minimal effect on the output's fluency and adequacy.
The design of expressive representations of entities and relations in a knowledge graph is an important endeavor. While many of the existing approaches have primarily focused on learning from relational patterns and structural information, the intrin
sic complexity of KG entities has been more or less overlooked. More concretely, we hypothesize KG entities may be more complex than we think, i.e., an entity may wear many hats and relational triplets may form due to more than a single reason. To this end, this paper proposes to learn disentangled representations of KG entities - a new method that disentangles the inner latent properties of KG entities. Our disentangled process operates at the graph level and a neighborhood mechanism is leveraged to disentangle the hidden properties of each entity. This disentangled representation learning approach is model agnostic and compatible with canonical KG embedding approaches. We conduct extensive experiments on several benchmark datasets, equipping a variety of models (DistMult, SimplE, and QuatE) with our proposed disentangling mechanism. Experimental results demonstrate that our proposed approach substantially improves performance on key metrics.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
