Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLet's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses
دعونا لعب Mono-Poly: Bert يمكن أن تكشف عن الكلمات "مستوى Polysemy وقابلية القابلية إلى الحواس
68
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
ترميز نماذج اللغة المدربة مسبقا (LMS) معلومات غنية حول الهيكل اللغوي ولكن معرفتهم حول Polysemy المعجمية لا تزال غير واضحة. نقترح إعداد تجريبي رواية لتحليل هذه المعرفة في LMS مدربين على وجه التحديد لغات مختلفة (الإنجليزية والفرنسية والإسبانية واليونانية) وفي بيرتف متعدد اللغات. نقوم بإجراء تحليلنا على مجموعات البيانات المصممة بعناية لتعكس التوزيعات المختلفة بالمعنى، والتحكم في المعلمات التي يرتبط بها بشدة مع Polysemy مثل التردد والفئة النحوية. نوضح أن تمثيلات بيرت المشتقة تعكس الكلمات "مستوى Polysemy وقابلية القسم الخاصة بهم إلى الحواس. تعد المعلومات المتعلقة بالبوليزمي موجودة بشكل أكثر وضوحا في English BertBeddings، لكن النماذج الموجودة بلغات أخرى تمكنت أيضا من إنشاء تمييزات ذات صلة بين الكلمات في مستويات Polysemy المختلفة. تسهم نتائجنا في فهم أفضل للمعارف المشفرة في تمثيلات السياق وفتح طرق جديدة للبحث عن دلالات المعجمات متعددة اللغات.
Pre-trained language models (LMs) encode rich information about linguistic structure but their knowledge about lexical polysemy remains unclear. We propose a novel experimental setup for analyzing this knowledge in LMs specifically trained for different languages (English, French, Spanish, and Greek) and in multilingual BERT. We perform our analysis on datasets carefully designed to reflect different sense distributions, and control for parameters that are highly correlated with polysemy such as frequency and grammatical category. We demonstrate that BERT-derived representations reflect words' polysemy level and their partitionability into senses. Polysemy-related information is more clearly present in English BERT embeddings, but models in other languages also manage to establish relevant distinctions between words at different polysemy levels. Our results contribute to a better understanding of the knowledge encoded in contextualized representations and open up new avenues for multilingual lexical semantics research.
References used
https://aclanthology.org/
rate research
Read More
Latent alignment objectives such as CTC and AXE significantly improve non-autoregressive machine translation models. Can they improve autoregressive models as well? We explore the possibility of training autoregressive machine translation models with
latent alignment objectives, and observe that, in practice, this approach results in degenerate models. We provide a theoretical explanation for these empirical results, and prove that latent alignment objectives are incompatible with teacher forcing.
Despite achieving encouraging results, neural Referring Expression Generation models are often thought to lack transparency. We probed neural Referential Form Selection (RFS) models to find out to what extent the linguistic features influencing the R
E form are learned and captured by state-of-the-art RFS models. The results of 8 probing tasks show that all the defined features were learned to some extent. The probing tasks pertaining to referential status and syntactic position exhibited the highest performance. The lowest performance was achieved by the probing models designed to predict discourse structure properties beyond the sentence level.
Sub-tasks of intent classification, such as robustness to distribution shift, adaptation to specific user groups and personalization, out-of-domain detection, require extensive and flexible datasets for experiments and evaluation. As collecting such
datasets is time- and labor-consuming, we propose to use text generation methods to gather datasets. The generator should be trained to generate utterances that belong to the given intent. We explore two approaches to the generation of task-oriented utterances: in the zero-shot approach, the model is trained to generate utterances from seen intents and is further used to generate utterances for intents unseen during training. In the one-shot approach, the model is presented with a single utterance from a test intent. We perform a thorough automatic, and human evaluation of the intrinsic properties of two-generation approaches. The attributes of the generated data are close to original test sets, collected via crowd-sourcing.
Vector representations have become a central element in semantic language modelling, leading to mathematical overlaps with many fields including quantum theory. Compositionality is a core goal for such representations: given representations for wet'
and fish', how should the concept wet fish' be represented? This position paper surveys this question from two points of view. The first considers the question of whether an explicit mathematical representation can be successful using only tools from within linear algebra, or whether other mathematical tools are needed. The second considers whether semantic vector composition should be explicitly described mathematically, or whether it can be a model-internal side-effect of training a neural network. A third and newer question is whether a compositional model can be implemented on a quantum computer. Given the fundamentally linear nature of quantum mechanics, we propose that these questions are related, and that this survey may help to highlight candidate operations for future quantum implementation.
To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just good enough'' in the context of imperfect QA datasets. We explore the use of natural language inference (NLI) as a way
to achieve this goal, as NLI inherently requires the premise (document context) to contain all necessary information to support the hypothesis (proposed answer to the question). We leverage large pre-trained models and recent prior datasets to construct powerful question conversion and decontextualization modules, which can reformulate QA instances as premise-hypothesis pairs with very high reliability. Then, by combining standard NLI datasets with NLI examples automatically derived from QA training data, we can train NLI models to evaluate QA models' proposed answers. We show that our approach improves the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. Careful manual analysis over the predictions of our NLI model shows that it can further identify cases where the QA model produces the right answer for the wrong reason, i.e., when the answer sentence cannot address all aspects of the question.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
