Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userDoes BERT Learn as Humans Perceive? Understanding Linguistic Styles through Lexica
هل تتعلم بيرت كإنسان؟فهم الأساليب اللغوية من خلال lexica
606
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
ينقل الناس نيتهم وموقفهم من خلال الأساليب اللغوية للنص الذي يكتبونه. في هذه الدراسة، نقوم بتحقيق كملات المعجم في المعجم عبر الأساليب طوال العدسين: الإدراك البشري وأهمية كلمة الجهاز، لأن الكلمات تختلف في قوة الإشارات الأسلوبية التي تقدمها. لجمع ملصقات التصور البشري، فإننا نرفع مجموعة بيانات جديدة وطنانيرد، على رأس مجموعات بيانات النمط القياسي. لدينا عمال الحشد يسلط الضوء على الكلمات التمثيلية في النص الذي يجعلهم يعتقدون أن النص لديه الأنماط التالية: المداراة والشعور والتهدفة وخمس أنواع العاطفة. بعد ذلك بمقارنة هذه الملصقات البشرية هذه ذات أهمية نصية مشتقة من مصنف ذو طراز ذو ضبط صقل شهير مثل بيرت. تظهر نتائجنا أن بيرتف غالبا ما يجد كلمات المحتوى غير ذات صلة بالأناقة المستهدفة ككلمات مهمة تستخدم في التنبؤ بالأناقة، لكن البشر لا ينظرون بنفس الطريقة على الرغم من أن بعض الأساليب (مثل الشعور والإيجابي والفرح) الإنسان والجهاز الكلمات المحددة تشترك في تداخل كبير لبعض الأساليب.
People convey their intention and attitude through linguistic styles of the text that they write. In this study, we investigate lexicon usages across styles throughout two lenses: human perception and machine word importance, since words differ in the strength of the stylistic cues that they provide. To collect labels of human perception, we curate a new dataset, Hummingbird, on top of benchmarking style datasets. We have crowd workers highlight the representative words in the text that makes them think the text has the following styles: politeness, sentiment, offensiveness, and five emotion types. We then compare these human word labels with word importance derived from a popular fine-tuned style classifier like BERT. Our results show that the BERT often finds content words not relevant to the target style as important words used in style prediction, but humans do not perceive the same way even though for some styles (e.g., positive sentiment and joy) human- and machine-identified words share significant overlap for some styles.
References used
https://aclanthology.org/
rate research
Read More
Understanding idioms is important in NLP. In this paper, we study to what extent pre-trained BERT model can encode the meaning of a potentially idiomatic expression (PIE) in a certain context. We make use of a few existing datasets and perform two pr
obing tasks: PIE usage classification and idiom paraphrase identification. Our experiment results suggest that BERT indeed can separate the literal and idiomatic usages of a PIE with high accuracy. It is also able to encode the idiomatic meaning of a PIE to some extent.
In machine reading comprehension tasks, a model must extract an answer from the available context given a question and a passage. Recently, transformer-based pre-trained language models have achieved state-of-the-art performance in several natural la
nguage processing tasks. However, it is unclear whether such performance reflects true language understanding. In this paper, we propose adversarial examples to probe an Arabic pre-trained language model (AraBERT), leading to a significant performance drop over four Arabic machine reading comprehension datasets. We present a layer-wise analysis for the transformer's hidden states to offer insights into how AraBERT reasons to derive an answer. The experiments indicate that AraBERT relies on superficial cues and keyword matching rather than text understanding. Furthermore, hidden state visualization demonstrates that prediction errors can be recognized from vector representations in earlier layers.
The paper describes the MilaNLP team's submission (Bocconi University, Milan) in the WASSA 2021 Shared Task on Empathy Detection and Emotion Classification. We focus on Track 2 - Emotion Classification - which consists of predicting the emotion of re
actions to English news stories at the essay-level. We test different models based on multi-task and multi-input frameworks. The goal was to better exploit all the correlated information given in the data set. We find, though, that empathy as an auxiliary task in multi-task learning and demographic attributes as additional input provide worse performance with respect to single-task learning. While the result is competitive in terms of the competition, our results suggest that emotion and empathy are not related tasks - at least for the purpose of prediction.
Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words -- either through masking or next sentence prediction -- and has no kn
owledge of lexical, syntactic or semantic information beyond what it picks up through unsupervised pre-training. We propose a novel method to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. When injecting counter-fitted and dependency-based embeddings, the performance improvements on multiple semantic similarity datasets indicate that such information is beneficial and currently missing from the original model. Our qualitative analysis shows that counter-fitted embedding injection is particularly beneficial, with notable improvements on examples that require synonym resolution.
The need to deploy large-scale pre-trained models on edge devices under limited computational resources has led to substantial research to compress these large models. However, less attention has been given to compress the task-specific models. In th
is work, we investigate the different methods of unstructured pruning on task-specific models for Aspect-based Sentiment Analysis (ABSA) tasks. Specifically, we analyze differences in the learning dynamics of pruned models by using the standard pruning techniques to achieve high-performing sparse networks. We develop a hypothesis to demonstrate the effectiveness of local pruning over global pruning considering a simple CNN model. Later, we utilize the hypothesis to demonstrate the efficacy of the pruned state-of-the-art model compared to the over-parameterized state-of-the-art model under two settings, the first considering the baselines for the same task used for generating the hypothesis, i.e., aspect extraction and the second considering a different task, i.e., sentiment analysis. We also provide discussion related to the generalization of the pruning hypothesis.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
