Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userExploring the Role of BERT Token Representations to Explain Sentence Probing Results
استكشاف دور تمثيلات Bert Token لشرح نتائج تحقيق الجملة
303
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تم تنفيذ العديد من الدراسات على الكشف عن الميزات اللغوية التي أسرتها بيرت. يتم تحقيق ذلك عادة من خلال تدريب مصنف تشخيصي على تمثيلات تم الحصول عليها من طبقات مختلفة من بيرت. ثم يتم تفسير دقة التصنيف اللاحقة على أنها قدرة النموذج في ترميز الممتلكات اللغوية المقابلة. على الرغم من تقديم رؤى، فقد تركت هذه الدراسات الدور المحتمل لتمثيلات الرمز المميز. في هذه الورقة، نقدم تحليلا أكثر متعمقا حول مساحة تمثيل بيرت بحثا عن مساحات فرعية متميزة وذات مغزى يمكن أن تفسر الأسباب الكامنة وراء هذه النتائج التحقيق. بناء على مجموعة من المهام التحقيق ومع مساعدة أساليب الإسناد، نوضح أن بيرت يميل إلى تشفير المعرفة الهادفة في تمثيلات رمزية محددة (والتي غالبا ما يتم تجاهلها في إعدادات التصنيف القياسية)، مما يسمح للنموذج بالكشف عن تشوهات النحوية والدلالية، ولل منفصلة بشكل مميز رقم النحوي والضواس الفرعية المتوترة.
Several studies have been carried out on revealing linguistic features captured by BERT. This is usually achieved by training a diagnostic classifier on the representations obtained from different layers of BERT. The subsequent classification accuracy is then interpreted as the ability of the model in encoding the corresponding linguistic property. Despite providing insights, these studies have left out the potential role of token representations. In this paper, we provide a more in-depth analysis on the representation space of BERT in search for distinct and meaningful subspaces that can explain the reasons behind these probing results. Based on a set of probing tasks and with the help of attribution methods we show that BERT tends to encode meaningful knowledge in specific token representations (which are often ignored in standard classification setups), allowing the model to detect syntactic and semantic abnormalities, and to distinctively separate grammatical number and tense subspaces.
References used
https://aclanthology.org/
rate research
Read More
Document-level event extraction is critical to various natural language processing tasks for providing structured information. Existing approaches by sequential modeling neglect the complex logic structures for long texts. In this paper, we leverage
the entity interactions and sentence interactions within long documents and transform each document into an undirected unweighted graph by exploiting the relationship between sentences. We introduce the Sentence Community to represent each event as a subgraph. Furthermore, our framework SCDEE maintains the ability to extract multiple events by sentence community detection using graph attention networks and alleviate the role overlapping issue by predicting arguments in terms of roles. Experiments demonstrate that our framework achieves competitive results over state-of-the-art methods on the large-scale document-level event extraction dataset.
Most question answering tasks focuses on predicting concrete answers, e.g., named entities. These tasks can be normally achieved by understanding the contexts without additional information required. In Reading Comprehension of Abstract Meaning (ReCA
M) task, the abstract answers are introduced. To understand abstract meanings in the context, additional knowledge is essential. In this paper, we propose an approach that leverages the pre-trained BERT Token embeddings as a prior knowledge resource. According to the results, our approach using the pre-trained BERT outperformed the baselines. It shows that the pre-trained BERT token embeddings can be used as additional knowledge for understanding abstract meanings in question answering.
In this paper, we propose to align sentence representations from different languages into a unified embedding space, where semantic similarities (both cross-lingual and monolingual) can be computed with a simple dot product. Pre-trained language mode
ls are fine-tuned with the translation ranking task. Existing work (Feng et al., 2020) uses sentences within the same batch as negatives, which can suffer from the issue of easy negatives. We adapt MoCo (He et al., 2020) to further improve the quality of alignment. As the experimental results show, the sentence representations produced by our model achieve the new state-of-the-art on several tasks, including Tatoeba en-zh similarity search (Artetxe andSchwenk, 2019b), BUCC en-zh bitext mining, and semantic textual similarity on 7 datasets.
Existing pre-trained language models (PLMs) are often computationally expensive in inference, making them impractical in various resource-limited real-world applications. To address this issue, we propose a dynamic token reduction approach to acceler
ate PLMs' inference, named TR-BERT, which could flexibly adapt the layer number of each token in inference to avoid redundant calculation. Specially, TR-BERT formulates the token reduction process as a multi-step token selection problem and automatically learns the selection strategy via reinforcement learning. The experimental results on several downstream NLP tasks show that TR-BERT is able to speed up BERT by 2-5 times to satisfy various performance demands. Moreover, TR-BERT can also achieve better performance with less computation in a suite of long-text tasks since its token-level layer number adaption greatly accelerates the self-attention operation in PLMs. The source code and experiment details of this paper can be obtained from https://github.com/thunlp/TR-BERT.
This paper revisits feature engineering approaches for predicting the complexity level of English words in a particular context using regression techniques. Our best submission to the Lexical Complexity Prediction (LCP) shared task was ranked 3rd out
of 48 systems for sub-task 1 and achieved Pearson correlation coefficients of 0.779 and 0.809 for single words and multi-word expressions respectively. The conclusion is that a combination of lexical, contextual and semantic features can still produce strong baselines when compared against human judgement.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
