Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userApplying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary
تطبيق حلاقة Occam في تحليل التبعية القائمة على المحولات: ما الذي يعمل، ما لا، وما هو ضروري حقا
265
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
أدت إدخال مذكرات الكلمات المحول المستندة إلى المحولات المدربين مسبقا إلى تحسينات كبيرة في دقة المحللين المستندة إلى الرسم البياني للأطر مثل التبعيات العالمية (UD). ومع ذلك، يختلف الأمر السابق في الأبعاد المختلفة، بما في ذلك اختيارهم لنماذج اللغة المدربة مسبقا وما إذا كانوا يستخدمون طبقات LSTM. مع تهدف إلى تحرير آثار هذه الخيارات وتحديد بنية بسيطة ولكنها قابلة للتطبيق على نطاق واسع، نقدم خطوات، ومحلل التبعية المستندة إلى الرسم البياني المعياري الجديد. باستخدام خطوات، نقوم بإجراء سلسلة من التحليلات على OD Corpora من مجموعة متنوعة من اللغات. نجد أن اختيار المدينات المدربة مسبقا له كبير تأثير على أداء المحلل وتحديد XLM-R كخيار قوي عبر اللغات في دراستنا. لا توفر إضافة طبقات LSTM أي فوائد عند استخدام Embeddings القائمة على المحولات. قد يؤدي إعداد إعدادات التدريب متعددة المهام إلى إخراج ميزات UD إضافية. أخذ هذه الأفكار معا، نقترح بنية ومحزين بسيطة ولكنها قابلة للتطبيق على نطاق واسع، وتحقيق نتائج جديدة من من من بين الفنون (من حيث LAS) لمدة 10 لغات مختلفة.
The introduction of pre-trained transformer-based contextualized word embeddings has led to considerable improvements in the accuracy of graph-based parsers for frameworks such as Universal Dependencies (UD). However, previous works differ in various dimensions, including their choice of pre-trained language models and whether they use LSTM layers. With the aims of disentangling the effects of these choices and identifying a simple yet widely applicable architecture, we introduce STEPS, a new modular graph-based dependency parser. Using STEPS, we perform a series of analyses on the UD corpora of a diverse set of languages. We find that the choice of pre-trained embeddings has by far the greatest impact on parser performance and identify XLM-R as a robust choice across the languages in our study. Adding LSTM layers provides no benefits when using transformer-based embeddings. A multi-task training setup outputting additional UD features may contort results. Taking these insights together, we propose a simple but widely applicable parser architecture and configuration, achieving new state-of-the-art results (in terms of LAS) for 10 out of 12 diverse languages.
References used
https://aclanthology.org/
rate research
Read More
In the last few years, several methods have been proposed to build meta-embeddings. The general aim was to obtain new representations integrating complementary knowledge from different source pre-trained embeddings thereby improving their overall qua
lity. However, previous meta-embeddings have been evaluated using a variety of methods and datasets, which makes it difficult to draw meaningful conclusions regarding the merits of each approach. In this paper we propose a unified common framework, including both intrinsic and extrinsic tasks, for a fair and objective meta-embeddings evaluation. Furthermore, we present a new method to generate meta-embeddings, outperforming previous work on a large number of intrinsic evaluation benchmarks. Our evaluation framework also allows us to conclude that previous extrinsic evaluations of meta-embeddings have been overestimated.
Entity Linking (EL) systems have achieved impressive results on standard benchmarks mainly thanks to the contextualized representations provided by recent pretrained language models. However, such systems still require massive amounts of data -- mill
ions of labeled examples -- to perform at their best, with training times that often exceed several days, especially when limited computational resources are available. In this paper, we look at how Named Entity Recognition (NER) can be exploited to narrow the gap between EL systems trained on high and low amounts of labeled data. More specifically, we show how and to what extent an EL system can benefit from NER to enhance its entity representations, improve candidate selection, select more effective negative samples and enforce hard and soft constraints on its output entities. We release our software -- code and model checkpoints -- at https://github.com/Babelscape/ner4el.
Dialogue summarization helps readers capture salient information from long conversations in meetings, interviews, and TV series. However, real-world dialogues pose a great challenge to current summarization models, as the dialogue length typically ex
ceeds the input limits imposed by recent transformer-based pre-trained models, and the interactive nature of dialogues makes relevant information more context-dependent and sparsely distributed than news articles. In this work, we perform a comprehensive study on long dialogue summarization by investigating three strategies to deal with the lengthy input problem and locate relevant information: (1) extended transformer models such as Longformer, (2) retrieve-then-summarize pipeline models with several dialogue utterance retrieval methods, and (3) hierarchical dialogue encoding models such as HMNet. Our experimental results on three long dialogue datasets (QMSum, MediaSum, SummScreen) show that the retrieve-then-summarize pipeline models yield the best performance. We also demonstrate that the summary quality can be further improved with a stronger retrieval model and pretraining on proper external summarization datasets.
The last years have shown rapid developments in the field of multimodal machine learning, combining e.g., vision, text or speech. In this position paper we explain how the field uses outdated definitions of multimodality that prove unfit for the mach
ine learning era. We propose a new task-relative definition of (multi)modality in the context of multimodal machine learning that focuses on representations and information that are relevant for a given machine learning task. With our new definition of multimodality we aim to provide a missing foundation for multimodal research, an important component of language grounding and a crucial milestone towards NLU.
Natural Language Processing tools and resources have been so far mainly created and trained for standard varieties of language. Nowadays, with the use of large amounts of data gathered from social media, other varieties and registers need to be proce
ssed, which may present other challenges and difficulties. In this work, we focus on English and we present a preliminary analysis by comparing the TwitterAAE corpus, which is annotated for ethnicity, and WordNet by quantifying and explaining the online language that WordNet misses.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
