Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userStrength in Numbers: Averaging and Clustering Effects in Mixture of Experts for Graph-Based Dependency Parsing
القوة في الأرقام: متوسط وتجميع تأثيرات التجميع في مزيج من الخبراء لتحليل التبعية القائمة على الرسم البياني
271
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نحن نراجع ميزتين من مزيج من نماذج الخبراء (MOE) التي نسميها وتأثيرات التجميع في سياق محلل التبعية القائمة على الرسوم البيانية المستفادة في إطار احتمامي إشراف. يتوافق المتوسط مع مزيج الفرقة من المحللين وهو مسؤول عن تخفيض التباين الذي يساعد على استقرار وتحسين دقة التحليل. يصف التجميع طاقة نماذج وزارة التعليم لتعطي المزيد من الائتمان للخبراء يعتقد أنه أكثر دقة بالنظر إلى المدخلات. على الرغم من الواعدة، يصعب تحقيق ذلك، خاصة دون بيانات إضافية. نقوم بتصميم إعداد تجريبي لدراسة تأثير هذه الآثار. في حين أن المتوسط مفيد دائما، فإن التجميع يتطلب تقنيات التهيئة والاستقرار جيدة، لكن مزاياها على مدى المتوسط يبدو أنها تختفي في نهاية المطاف عندما يوجد عدد كاف من الخبراء. كمنتج حسب المنتج، نوضح كيف يؤدي ذلك إلى نتائج أحدث النتائج على PTB و Conll09 Treebank الصينية، مع انخفاض التباين عبر التجارب.
We review two features of mixture of experts (MoE) models which we call averaging and clustering effects in the context of graph-based dependency parsers learned in a supervised probabilistic framework. Averaging corresponds to the ensemble combination of parsers and is responsible for variance reduction which helps stabilizing and improving parsing accuracy. Clustering describes the capacity of MoE models to give more credit to experts believed to be more accurate given an input. Although promising, this is difficult to achieve, especially without additional data. We design an experimental set-up to study the impact of these effects. Whereas averaging is always beneficial, clustering requires good initialization and stabilization techniques, but its advantages over mere averaging seem to eventually vanish when enough experts are present. As a by product, we show how this leads to state-of-the-art results on the PTB and the CoNLL09 Chinese treebank, with low variance across experiments.
References used
https://aclanthology.org/
rate research
Read More
Abstract Interpretable rationales for model predictions are crucial in practical applications. We develop neural models that possess an interpretable inference process for dependency parsing. Our models adopt instance-based inference, where dependenc
y edges are extracted and labeled by comparing them to edges in a training set. The training edges are explicitly used for the predictions; thus, it is easy to grasp the contribution of each edge to the predictions. Our experiments show that our instance-based models achieve competitive accuracy with standard neural models and have the reasonable plausibility of instance-based explanations.
The dominant paradigm for semantic parsing in recent years is to formulate parsing as a sequence-to-sequence task, generating predictions with auto-regressive sequence decoders. In this work, we explore an alternative paradigm. We formulate semantic
parsing as a dependency parsing task, applying graph-based decoding techniques developed for syntactic parsing. We compare various decoding techniques given the same pre-trained Transformer encoder on the TOP dataset, including settings where training data is limited or contains only partially-annotated examples. We find that our graph-based approach is competitive with sequence decoders on the standard setting, and offers significant improvements in data efficiency and settings where partially-annotated data is available.
We propose the Recursive Non-autoregressive Graph-to-Graph Transformer architecture (RNGTr) for the iterative refinement of arbitrary graphs through the recursive application of a non-autoregressive Graph-to-Graph Transformer and apply it to syntacti
c dependency parsing. We demonstrate the power and effectiveness of RNGTr on several dependency corpora, using a refinement model pre-trained with BERT. We also introduce Syntactic Transformer (SynTr), a non-recursive parser similar to our refinement model. RNGTr can improve the accuracy of a variety of initial parsers on 13 languages from the Universal Dependencies Treebanks, English and Chinese Penn Treebanks, and the German CoNLL2009 corpus, even improving over the new state-of-the-art results achieved by SynTr, significantly improving the state-of-the-art for all corpora tested.
AM dependency parsing is a method for neural semantic graph parsing that exploits the principle of compositionality. While AM dependency parsers have been shown to be fast and accurate across several graphbanks, they require explicit annotations of t
he compositional tree structures for training. In the past, these were obtained using complex graphbank-specific heuristics written by experts. Here we show how they can instead be trained directly on the graphs with a neural latent-variable model, drastically reducing the amount and complexity of manual heuristics. We demonstrate that our model picks up on several linguistic phenomena on its own and achieves comparable accuracy to supervised training, greatly facilitating the use of AM dependency parsing for new sembanks.
Semantic representation that supports the choice of an appropriate connective between pairs of clauses inherently addresses discourse coherence, which is important for tasks such as narrative understanding, argumentation, and discourse parsing. We pr
opose a novel clause embedding method that applies graph learning to a data structure we refer to as a dependency-anchor graph. The dependency anchor graph incorporates two kinds of syntactic information, constituency structure, and dependency relations, to highlight the subject and verb phrase relation. This enhances coherence-related aspects of representation. We design a neural model to learn a semantic representation for clauses from graph convolution over latent representations of the subject and verb phrase. We evaluate our method on two new datasets: a subset of a large corpus where the source texts are published novels, and a new dataset collected from students' essays. The results demonstrate a significant improvement over tree-based models, confirming the importance of emphasizing the subject and verb phrase. The performance gap between the two datasets illustrates the challenges of analyzing student's written text, plus a potential evaluation task for coherence modeling and an application for suggesting revisions to students.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
