Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userBayesian Model-Agnostic Meta-Learning with Matrix-Valued Kernels for Quality Estimation
التعلم Meta النموذجي النموذجي مع النواة ذات القيمة المصفوفة لتقدير الجودة
145
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يتم تدريب معظم نماذج تقدير الجودة الحالية (QE) للترجمة الآلية وتقييمها في بيئة إشراف بالكامل تتطلب كميات كبيرة من بيانات التدريب المسمى. ومع ذلك، يمكن أن تكون الحصول على البيانات المسمى باهظة الثمن وتستغرق وقتا طويلا. بالإضافة إلى ذلك، قد تتعرض بيانات الاختبار التي سيتم التعرض لها نموذج QE المنشور قد تختلف عن بيانات التدريب الخاصة بها بطرق مهمة. على وجه الخصوص، غالبا ما يتم تصنيف عينات التدريب من خلال مجموعة واحدة أو مجموعة صغيرة من المعلقين، والتي قد تختلف تصورات جودة الترجمة واحتياجاتها بشكل كبير من هؤلاء المستخدمين النهائيين، الذين سيعملون التنبؤات في الممارسة العملية. وبالتالي، من المرغوب فيه أن تكون قادرا على التكيف مع نماذج QE بكفاءة إلى بيانات المستخدم الجديدة مع بيانات الإشراف المحدودة. لمعالجة هذه التحديات، نقترح نهجا لتعليم التعلم بايزي لتكييف نماذج QE لاحتياجات وتفضيلات كل مستخدم مع إشراف محدود. لتعزيز الأداء، فإننا نقترح كذلك امتدادا إلى نهج التعلم التلوي بايز بين الفنادق التي تستخدم نواة مصفوفة ذات قيمة لتعلم التلوي بايزي لتقدير الجودة. توضح التجارب المعنية ببيانات عدد متفاوت من المستخدمين والخصائص اللغوية أن نهج التعلم التلوي المقترح يقدم أداء تنبؤي محسن في كل من إعدادات الإشراف المحدودة والكامل.
Most current quality estimation (QE) models for machine translation are trained and evaluated in a fully supervised setting requiring significant quantities of labelled training data. However, obtaining labelled data can be both expensive and time-consuming. In addition, the test data that a deployed QE model would be exposed to may differ from its training data in significant ways. In particular, training samples are often labelled by one or a small set of annotators, whose perceptions of translation quality and needs may differ substantially from those of end-users, who will employ predictions in practice. Thus, it is desirable to be able to adapt QE models efficiently to new user data with limited supervision data. To address these challenges, we propose a Bayesian meta-learning approach for adapting QE models to the needs and preferences of each user with limited supervision. To enhance performance, we further propose an extension to a state-of-the-art Bayesian meta-learning approach which utilizes a matrix-valued kernel for Bayesian meta-learning of quality estimation. Experiments on data with varying number of users and language characteristics demonstrates that the proposed Bayesian meta-learning approach delivers improved predictive performance in both limited and full supervision settings.
References used
https://aclanthology.org/
rate research
Read More
This paper describes POSTECH's quality estimation systems submitted to Task 2 of the WMT 2021 quality estimation shared task: Word and Sentence-Level Post-editing Effort. We notice that it is possible to improve the stability of the latest quality es
timation models that have only one encoder based on the self-attention mechanism to simultaneously process both of the two input data, a source sequence and its machine translation, in that such models have neglected to take advantage of pre-trained monolingual representations, which are generally accepted as reliable representations for various natural language processing tasks. Therefore, our model uses two pre-trained monolingual encoders and then exchanges the information of two encoded representations through two additional cross attention networks. According to the official leaderboard, our systems outperform the baseline systems in terms of the Matthews correlation coefficient for machine translations' word-level quality estimation and in terms of the Pearson's correlation coefficient for sentence-level quality estimation by 0.4126 and 0.5497 respectively.
We propose a novel scheme to use the Levenshtein Transformer to perform the task of word-level quality estimation. A Levenshtein Transformer is a natural fit for this task: trained to perform decoding in an iterative manner, a Levenshtein Transformer
can learn to post-edit without explicit supervision. To further minimize the mismatch between the translation task and the word-level QE task, we propose a two-stage transfer learning procedure on both augmented data and human post-editing data. We also propose heuristics to construct reference labels that are compatible with subword-level finetuning and inference. Results on WMT 2020 QE shared task dataset show that our proposed method has superior data efficiency under the data-constrained setting and competitive performance under the unconstrained setting.
High-quality arguments are an essential part of decision-making. Automatically predicting the quality of an argument is a complex task that recently got much attention in argument mining. However, the annotation effort for this task is exceptionally
high. Therefore, we test uncertainty-based active learning (AL) methods on two popular argument-strength data sets to estimate whether sample-efficient learning can be enabled. Our extensive empirical evaluation shows that uncertainty-based acquisition functions can not surpass the accuracy reached with the random acquisition on these data sets.
Content moderation is often performed by a collaboration between humans and machine learning models. However, it is not well understood how to design the collaborative process so as to maximize the combined moderator-model system performance. This wo
rk presents a rigorous study of this problem, focusing on an approach that incorporates model uncertainty into the collaborative process. First, we introduce principled metrics to describe the performance of the collaborative system under capacity constraints on the human moderator, quantifying how efficiently the combined system utilizes human decisions. Using these metrics, we conduct a large benchmark study evaluating the performance of state-of-the-art uncertainty models under different collaborative review strategies. We find that an uncertainty-based strategy consistently outperforms the widely used strategy based on toxicity scores, and moreover that the choice of review strategy drastically changes the overall system performance. Our results demonstrate the importance of rigorous metrics for understanding and developing effective moderator-model systems for content moderation, as well as the utility of uncertainty estimation in this domain.
The combination of multilingual pre-trained representations and cross-lingual transfer learning is one of the most effective methods for building functional NLP systems for low-resource languages. However, for extremely low-resource languages without
large-scale monolingual corpora for pre-training or sufficient annotated data for fine-tuning, transfer learning remains an understudied and challenging task. Moreover, recent work shows that multilingual representations are surprisingly disjoint across languages, bringing additional challenges for transfer onto extremely low-resource languages. In this paper, we propose MetaXL, a meta-learning based framework that learns to transform representations judiciously from auxiliary languages to a target one and brings their representation spaces closer for effective transfer. Extensive experiments on real-world low-resource languages -- without access to large-scale monolingual corpora or large amounts of labeled data -- for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach. Code for MetaXL is publicly available at github.com/microsoft/MetaXL.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
