Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userAugmenting BERT-style Models with Predictive Coding to Improve Discourse-level Representations
زيادة نماذج نمط بيرت مع ترميز تنبؤي لتحسين تمثيلات مستوى الخطاب
339
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
عادة ما يتم تدريب نماذج اللغات الحالية على استخدام مخطط للإشراف على الذات، حيث يركز التركيز الرئيسي على التعلم في كلمة البرنامج أو مستوى الجملة.ومع ذلك، كان هناك تقدم محدود في توليد تمثيلات مفيدة على مستوى الخطاب.في هذا العمل، نقترح استخدام الأفكار من نظرية الترميز التنبؤية لزيادة نماذج اللغة ذات طراز بيرت مع آلية تسمح لهم بتعلم تمثيلات مناسبة على مستوى الخطاب.نتيجة لذلك، يكون نهجنا المقترح قادرا على التنبؤ بالأحكام المستقبلية باستخدام اتصالات واضحة من أعلى إلى أسفل تعمل في الطبقات المتوسطة للشبكة.من خلال تجربة معايير مصممة لتقييم المعرفة المتعلقة بالحبال باستخدام تمثيلات الجملة المدربة مسبقا، نوضح أن نهجنا يحسن الأداء في 6 من أصل 11 مهام من خلال التميز في كشف علاقة الخطاب.
Current language models are usually trained using a self-supervised scheme, where the main focus is learning representations at the word or sentence level. However, there has been limited progress in generating useful discourse-level representations. In this work, we propose to use ideas from predictive coding theory to augment BERT-style language models with a mechanism that allows them to learn suitable discourse-level representations. As a result, our proposed approach is able to predict future sentences using explicit top-down connections that operate at the intermediate layers of the network. By experimenting with benchmarks designed to evaluate discourse-related knowledge using pre-trained sentence representations, we demonstrate that our approach improves performance in 6 out of 11 tasks by excelling in discourse relationship detection.
References used
https://aclanthology.org/
rate research
Read More
Learning a good latent representation is essential for text style transfer, which generates a new sentence by changing the attributes of a given sentence while preserving its content. Most previous works adopt disentangled latent representation learn
ing to realize style transfer. We propose a novel text style transfer algorithm with entangled latent representation, and introduce a style classifier that can regulate the latent structure and transfer style. Moreover, our algorithm for style transfer applies to both single-attribute and multi-attribute transfer. Extensive experimental results show that our method generally outperforms state-of-the-art approaches.
The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations. In this work, we consider a new question: to what extent contextual representations of concrete nouns are aligned
with corresponding visual representations? We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Moreover, they are effective in retrieving specific instances of image patches; textual context plays an important role in this process. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans. We hope our analyses inspire future research in understanding and improving the visual capabilities of language models.
Discourse analysis has long been known to be fundamental in natural language processing. In this research, we present our insight on discourse-level topic chain (DTC) parsing which aims at discovering new topics and investigating how these topics evo
lve over time within an article. To address the lack of data, we contribute a new discourse corpus with DTC-style dependency graphs annotated upon news articles. In particular, we ensure the high reliability of the corpus by utilizing a two-step annotation strategy to build the data and filtering out the annotations with low confidence scores. Based on the annotated corpus, we introduce a simple yet robust system for automatic discourse-level topic chain parsing.
This paper describes our approach (ur-iw-hnt) for the Shared Task of GermEval2021 to identify toxic, engaging, and fact-claiming comments. We submitted three runs using an ensembling strategy by majority (hard) voting with multiple different BERT mod
els of three different types: German-based, Twitter-based, and multilingual models. All ensemble models outperform single models, while BERTweet is the winner of all individual models in every subtask. Twitter-based models perform better than GermanBERT models, and multilingual models perform worse but by a small margin.
This paper describes the HEL-LJU submissions to the MultiLexNorm shared task on multilingual lexical normalization. Our system is based on a BERT token classification preprocessing step, where for each token the type of the necessary transformation i
s predicted (none, uppercase, lowercase, capitalize, modify), and a character-level SMT step where the text is translated from original to normalized given the BERT-predicted transformation constraints. For some languages, depending on the results on development data, the training data was extended by back-translating OpenSubtitles data. In the final ordering of the ten participating teams, the HEL-LJU team has taken the second place, scoring better than the previous state-of-the-art.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
