Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userBLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-Oriented Text
بلو، نيزك، Bertscore: تقييم أداء المقاييس في تقييم أخطاء الترجمة الهامة في النص الموجه نحو المعنويات
450
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
شركات وسائل التواصل الاجتماعي وكذلك سلطات الرقابة تجعل الاستخدام المكثف للأدوات الذكاء الاصطناعي (AI) لمراقبة منشورات خطاب الكراهية أو الاحتفالات بالعنف أو الألفاظ النابية. نظرا لأن برنامج AI يتطلب كميات كبيرة من البيانات لتدريب أجهزة الكمبيوتر، يتم تنفيذ الترجمة التلقائية للمحتوى عبر الإنترنت عادة للتعويض عن ندرة النص في بعض اللغات. ومع ذلك، فإن أخطاء الترجمة الآلية (MT) هي حدوث منتظم عند ترجمة المحتوى الذي تم إنشاؤه الموجه نحو المستخدم (UGC)، خاصة عند مشاركة لغة الموارد المنخفضة. في مثل هذه السيناريوهات، تعتمد كفاية العملية برمتها على افتراض أن الترجمة يمكن تقييمها بشكل صحيح. في هذه الورقة، نقوم بتقييم قدرة مقاييس الجودة التلقائية على اكتشاف أخطاء ترجمة الآلات الحرجة التي يمكن أن تسبب سوء فهم خطير في رسالة التأثير. نقارن أداء ثلاثة مقاييس قانوني على الترجمات التي لا معنى لها بالمقارنة مع الترجمات ذات مغزى مع وجود خطأ حاسم يشوه المعنويات الإجمالية للنص المصدر. نوضح الحاجة إلى ضبط القيود التلقائي لجعلها أكثر قوة في اكتشاف أخطاء المعنويات الحرجة.
Social media companies as well as censorship authorities make extensive use of artificial intelligence (AI) tools to monitor postings of hate speech, celebrations of violence or profanity. Since AI software requires massive volumes of data to train computers, automatic-translation of the online content is usually implemented to compensate for the scarcity of text in some languages. However, machine translation (MT) mistakes are a regular occurrence when translating sentiment-oriented user-generated content (UGC), especially when a low-resource language is involved. In such scenarios, the adequacy of the whole process relies on the assumption that the translation can be evaluated correctly. In this paper, we assess the ability of automatic quality metrics to detect critical machine translation errors which can cause serious misunderstanding of the affect message. We compare the performance of three canonical metrics on meaningless translations as compared to meaningful translations with a critical error that distorts the overall sentiment of the source text. We demonstrate the need for the fine-tuning of automatic metrics to make them more robust in detecting sentiment critical errors.
References used
https://aclanthology.org/
rate research
Read More
This paper discusses a classification-based approach to machine translation evaluation, as opposed to a common regression-based approach in the WMT Metrics task. Recent machine translation usually works well but sometimes makes critical errors due to
just a few wrong word choices. Our classification-based approach focuses on such errors using several error type labels, for practical machine translation evaluation in an age of neural machine translation. We made additional annotations on the WMT 2015-2017 Metrics datasets with fluency and adequacy labels to distinguish different types of translation errors from syntactic and semantic viewpoints. We present our human evaluation criteria for the corpus development and automatic evaluation experiments using the corpus. The human evaluation corpus will be publicly available upon publication.
Many modern machine translation evaluation metrics like BERTScore, BLEURT, COMET, MonoTransquest or XMoverScore are based on black-box language models. Hence, it is difficult to explain why these metrics return certain scores. This year's Eval4NLP sh
ared task tackles this challenge by searching for methods that can extract feature importance scores that correlate well with human word-level error annotations. In this paper we show that unsupervised metrics that are based on tokenmatching can intrinsically provide such scores. The submitted system interprets the similarities of the contextualized word-embeddings that are used to compute (X)BERTScore as word-level importance scores.
Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains. It has been recently shown that the probabilities given by a large, multilingual
model can achieve state of the art results when used as a reference-free metric. We experiment with various modifications to this model, and demonstrate that by scaling it up we can match the performance of BLEU. We analyze various potential weaknesses of the approach, and find that it is surprisingly robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities.
Automated source code summarization is a popular software engineering research topic wherein machine translation models are employed to translate'' code snippets into relevant natural language descriptions. Most evaluations of such models are conduct
ed using automatic reference-based metrics. However, given the relatively large semantic gap between programming languages and natural language, we argue that this line of research would benefit from a qualitative investigation into the various error modes of current state-of-the-art models. Therefore, in this work, we perform both a quantitative and qualitative comparison of three recently proposed source code summarization models. In our quantitative evaluation, we compare the models based on the smoothed BLEU-4, METEOR, and ROUGE-L machine translation metrics, and in our qualitative evaluation, we perform a manual open-coding of the most common errors committed by the models when compared to ground truth captions. Our investigation reveals new insights into the relationship between metric-based performance and model prediction errors grounded in an error taxonomy that can be used to drive future research efforts.
The location of wireless sensor nodes located in the center is necessary for applications
where information about the site is important information such as security, protection,
object tracking and other applications.
localization algorithms are c
lassified into two types: Range-based and Range-free. The
study focused on Range-free localization algorithms because they are less expensive in
terms of hardware requirements.
The MATLAB was used to simulate the algorithms, whose performance was evaluated by
changing the number of network nodes, the number of Anchor nodes, and the contract area
of communication in order to illustrate performance differences in terms of localization
error.
The results showed the superiority of the amorphous algorithm, achieving high localization
accuracy and lower cost for the number of Anchor nodes needed to achieve a small error.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
