Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userSentiment-Aware Measure (SAM) for Evaluating Sentiment Transfer by Machine Translation Systems
قياس المعنويات (SAM) لتقييم نقل المشاعر بواسطة أنظمة الترجمة الآلية
213
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
في ترجمة النص حيث تعتبر المشاعر الرسالة الرئيسية، يعطي المترجمون البشريون اهتماما خاصا للكلمات تحمل المعنويات. السبب هو أن ترجمة غير صحيحة لهذه الكلمات سوف تفوت الجانب الأساسي للنص المصدر، أي شعور المؤلف. في العالم عبر الإنترنت، تستخدم أنظمة MT على نطاق واسع لترجمة المحتوى الذي تم إنشاؤه بواسطة المستخدم (UGC) مثل المراجعات، وتغريدات، ووظائف وسائل التواصل الاجتماعي، حيث تكون الرسالة الرئيسية في كثير من الأحيان موقف المؤلف الإيجابي أو السلبي تجاه موضوع النص. من المهم في مثل هذه السيناريوهات لقياس بدقة إلى حد ما يمكن أن يكون نظام MT أداة مساعدة واقعية موثوقة في نقل الرسالة الصحيحة. تتناول هذه الورقة مشكلة أقل معترف بها في مجال تقييم الترجمة الآلية التي تهم إلى أي مدى يتفق المقاييس التلقائية مع مستوى الذهب من التقييم البشري للحصول على ترجمة صحيحة للمشاعر. نقوم بتقييم فعالية مقاييس الجودة التقليدية في اكتشاف عدم فهم الثقة، خاصة عندما يكون الخطأ الوحيد في إخراج MT. نقترح قياس المعنويات العددية "تقييس" المناسب لتقييم دقة الرسالة المترجمة تؤثر في نص UGC بواسطة نظام MT. سنظهر أن دمج هذا التدبير على دراية المعنويات يمكن أن يعزز بشكل كبير ارتباط بعض مقاييس الجودة المتاحة مع الحكم الإنساني لترجمة دقيقة للمشاعر.
In translating text where sentiment is the main message, human translators give particular attention to sentiment-carrying words. The reason is that an incorrect translation of such words would miss the fundamental aspect of the source text, i.e. the author's sentiment. In the online world, MT systems are extensively used to translate User-Generated Content (UGC) such as reviews, tweets, and social media posts, where the main message is often the author's positive or negative attitude towards the topic of the text. It is important in such scenarios to accurately measure how far an MT system can be a reliable real-life utility in transferring the correct affect message. This paper tackles an under-recognized problem in the field of machine translation evaluation which is judging to what extent automatic metrics concur with the gold standard of human evaluation for a correct translation of sentiment. We evaluate the efficacy of conventional quality metrics in spotting a mistranslation of sentiment, especially when it is the sole error in the MT output. We propose a numerical sentiment-closeness'' measure appropriate for assessing the accuracy of a translated affect message in UGC text by an MT system. We will show that incorporating this sentiment-aware measure can significantly enhance the correlation of some available quality metrics with the human judgement of an accurate translation of sentiment.
References used
https://aclanthology.org/
rate research
Read More
This paper describes NiuTrans neural machine translation systems of the WMT 2021 news translation tasks. We made submissions to 9 language directions, including English2Chinese, Japanese, Russian, Icelandic and English2Hausa tasks. Our primary system
s are built on several effective variants of Transformer, e.g., Transformer-DLCL, ODE-Transformer. We also utilize back-translation, knowledge distillation, post-ensemble, and iterative fine-tuning techniques to enhance the model performance further.
This paper describes our system (Team ID: nictrb) for participating in the WAT'21 restricted machine translation task. In our submitted system, we designed a new training approach for restricted machine translation. By sampling from the translation t
arget, we can solve the problem that ordinary training data does not have a restricted vocabulary. With the further help of constrained decoding in the inference phase, we achieved better results than the baseline, confirming the effectiveness of our solution. In addition, we also tried the vanilla and sparse Transformer as the backbone network of the model, as well as model ensembling, which further improved the final translation performance.
Aspect-based sentiment analysis (ABSA) mainly involves three subtasks: aspect term extraction, opinion term extraction, and aspect-level sentiment classification, which are typically handled in a separate or joint manner. However, previous approaches
do not well exploit the interactive relations among three subtasks and do not pertinently leverage the easily available document-level labeled domain/sentiment knowledge, which restricts their performances. To address these issues, we propose a novel Iterative Multi-Knowledge Transfer Network (IMKTN) for end-to-end ABSA. For one thing, through the interactive correlations between the ABSA subtasks, our IMKTN transfers the task-specific knowledge from any two of the three subtasks to another one at the token level by utilizing a well-designed routing algorithm, that is, any two of the three subtasks will help the third one. For another, our IMKTN pertinently transfers the document-level knowledge, i.e., domain-specific and sentiment-related knowledge, to the aspect-level subtasks to further enhance the corresponding performance. Experimental results on three benchmark datasets demonstrate the effectiveness and superiority of our approach.
This work introduces a simple regressive ensemble for evaluating machine translation quality based on a set of novel and established metrics. We evaluate the ensemble using a correlation to expert-based MQM scores of the WMT 2021 Metrics workshop. In
both monolingual and zero-shot cross-lingual settings, we show a significant performance improvement over single metrics. In the cross-lingual settings, we also demonstrate that an ensemble approach is well-applicable to unseen languages. Furthermore, we identify a strong reference-free baseline that consistently outperforms the commonly-used BLEU and METEOR measures and significantly improves our ensemble's performance.
In this work, two Neural Machine Translation (NMT) systems have been developed and evaluated as part of the bidirectional Tamil-Telugu similar languages translation subtask in WMT21. The OpenNMT-py toolkit has been used to create quick prototypes of
the systems, following which models have been trained on the training datasets containing the parallel corpus and finally the models have been evaluated on the dev datasets provided as part of the task. Both the systems have been trained on a DGX station with 4 -V100 GPUs. The first NMT system in this work is a Transformer based 6 layer encoder-decoder model, trained for 100000 training steps, whose configuration is similar to the one provided by OpenNMT-py and this is used to create a model for bidirectional translation. The second NMT system contains two unidirectional translation models with the same configuration as the first system, with the addition of utilizing Byte Pair Encoding (BPE) for subword tokenization through the pre-trained MultiBPEmb model. Based on the dev dataset evaluation metrics for both the systems, the first system i.e. the vanilla Transformer model has been submitted as the Primary system. Since there were no improvements in the metrics during training of the second system with BPE, it has been submitted as a contrastive system.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
