Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userDisentangling Representations of Text by Masking Transformers
منح تمثيل النص من خلال إخفاء المحولات
240
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تمثل التمثيلات من النماذج الكبيرة المحددة مسبقا مثل Bert مجموعة من الميزات في متجهات غير متجانسة، مما يوفر دقة تنبؤية قوية عبر مجموعة من المهام المصب. في هذه الورقة، نستكشف ما إذا كان من الممكن تعلم تمثيلات محددة من خلال تحديد الشبكات الفرعية الحالية داخل النماذج المحددة التي ترمز الجوانب المميزة والكمية. بشكل ملموس، نتعلم الأقنعة الثنائية على أوزان المحولات أو الوحدات المخفية للكشف عن مجموعات فرعية من الميزات التي ترتبط مع عامل اختلاف محدد؛ هذا يلغي الحاجة إلى تدريب نموذج DESTANGLED من الصفر لمهمة معينة. نقوم بتقييم هذه الطريقة فيما يتعلق بقدرتها على تحفيز تمثيلات المعنويات من النوع في مراجعات الأفلام، السمية من لهجة التغريدات، وبناء جملة من دلالات. من خلال الجمع بين التقليب مع تقليم الحجم، نجد أنه يمكننا تحديد الشبكات الفرعية المتنينة في بيرت التي ترمز بقوة جوانب معينة (على سبيل المثال، الدلالات) في حين ترميز الآخرين ضعيفة (E.G.، بناء الجملة). علاوة على ذلك، على الرغم من الأقنعة التعليمية فقط، فإن الإعفاء من عمليات الإخفاء فقط - - - وغالبا ما تكون أفضل من الأساليب المقترحة سابقا استنادا إلى السيارات الآلية المتنوعة والتدريب الخصم.
Representations from large pretrained models such as BERT encode a range of features into monolithic vectors, affording strong predictive accuracy across a range of downstream tasks. In this paper we explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects. Concretely, we learn binary masks over transformer weights or hidden units to uncover subsets of features that correlate with a specific factor of variation; this eliminates the need to train a disentangled model from scratch for a particular task. We evaluate this method with respect to its ability to disentangle representations of sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics. By combining masking with magnitude pruning we find that we can identify sparse subnetworks within BERT that strongly encode particular aspects (e.g., semantics) while only weakly encoding others (e.g., syntax). Moreover, despite only learning masks, disentanglement-via-masking performs as well as --- and often better than ---previously proposed methods based on variational autoencoders and adversarial training.
References used
https://aclanthology.org/
rate research
Read More
Being able to accurately perform Question Difficulty Estimation (QDE) can improve the accuracy of students' assessment and better their learning experience. Traditional approaches to QDE are either subjective or introduce a long delay before new ques
tions can be used to assess students. Thus, recent work proposed machine learning-based approaches to overcome these limitations. They use questions of known difficulty to train models capable of inferring the difficulty of questions from their text. Once trained, they can be used to perform QDE of newly created questions. Existing approaches employ supervised models which are domain-dependent and require a large dataset of questions of known difficulty for training. Therefore, they cannot be used if such a dataset is not available ( for new courses on an e-learning platform). In this work, we experiment with the possibility of performing QDE from text in an unsupervised manner. Specifically, we use the uncertainty of calibrated question answering models as a proxy of human-perceived difficulty. Our experiments show promising results, suggesting that model uncertainty could be successfully leveraged to perform QDE from text, reducing both costs and elapsed time.
We propose neural models to generate text from formal meaning representations based on Discourse Representation Structures (DRSs). DRSs are document-level representations which encode rich semantic detail pertaining to rhetorical relations, presuppos
ition, and co-reference within and across sentences. We formalize the task of neural DRS-to-text generation and provide modeling solutions for the problems of condition ordering and variable naming which render generation from DRSs non-trivial. Our generator relies on a novel sibling treeLSTM model which is able to accurately represent DRS structures and is more generally suited to trees with wide branches. We achieve competitive performance (59.48 BLEU) on the GMB benchmark against several strong baselines.
We present an end-to-end neural approach to generate English sentences from formal meaning representations, Discourse Representation Structures (DRSs). We use a rather standard bi-LSTM sequence-to-sequence model, work with a linearized DRS input repr
esentation, and evaluate character-level and word-level decoders. We obtain very encouraging results in terms of reference-based automatic metrics such as BLEU. But because such metrics only evaluate the surface level of generated output, we develop a new metric, ROSE, that targets specific semantic phenomena. We do this with five DRS generation challenge sets focusing on tense, grammatical number, polarity, named entities and quantities. The aim of these challenge sets is to assess the neural generator's systematicity and generalization to unseen inputs.
In Visual Question Answering (VQA), existing bilinear methods focus on the interaction between images and questions. As a result, the answers are either spliced into the questions or utilized as labels only for classification. On the other hand, tril
inear models such as the CTI model efficiently utilize the inter-modality information between answers, questions, and images, while ignoring intra-modality information. Inspired by this observation, we propose a new trilinear interaction framework called MIRTT (Learning Multimodal Interaction Representations from Trilinear Transformers), incorporating the attention mechanisms for capturing inter-modality and intra-modality relationships. Moreover, we design a two-stage workflow where a bilinear model reduces the free-form, open-ended VQA problem into a multiple-choice VQA problem. Furthermore, to obtain accurate and generic multimodal representations, we pre-train MIRTT with masked language prediction. Our method achieves state-of-the-art performance on the Visual7W Telling task and VQA-1.0 Multiple Choice task and outperforms bilinear baselines on the VQA-2.0, TDIUC and GQA datasets.
Current NLP models are predominantly trained through a two-stage pre-train then fine-tune'' pipeline. Prior work has shown that inserting an intermediate pre-training stage, using heuristic masking policies for masked language modeling (MLM), can sig
nificantly improve final performance. However, it is still unclear (1) in what cases such intermediate pre-training is helpful, (2) whether hand-crafted heuristic objectives are optimal for a given task, and (3) whether a masking policy designed for one task is generalizable beyond that task. In this paper, we perform a large-scale empirical study to investigate the effect of various masking policies in intermediate pre-training with nine selected tasks across three categories. Crucially, we introduce methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. We conclude that the success of intermediate pre-training is dependent on appropriate pre-train corpus, selection of output format (i.e., masked spans or full sentence), and clear understanding of the role that MLM plays for the downstream task. In addition, we find our learned masking policies outperform the heuristic of masking named entities on TriviaQA, and policies learned from one task can positively transfer to other tasks in certain cases, inviting future research in this direction.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
