Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userSentSim: Crosslingual Semantic Evaluation of Machine Translation
Sesstim: التقييم الدلالي crosslingual للترجمة الآلية
128
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تم تقييم الترجمة الآلية (MT) حاليا بأحد طريقتين: بطريقة أحادية الأجل، بالمقارنة مع إخراج النظام إلى ترجمات مرجعية بشرية واحدة أو أكثر، أو في أزياء تقاطعات مدربة، من خلال بناء نموذج إشرافي للتنبؤ بعشرات الجودة من البيانات ذات العلامات على الإنسان. في هذه الورقة، نقترح بديل أكثر فعالية من حيث التكلفة، ومع ذلك أدائين جيدا من حيث التكلفة، والاعتماد على كلمة متعددة المحاكمات متعددة اللغات وتمثيلات الجملة، ونحن نقارن مباشرة المصدر مع الجملة ترجمت الآلة، وبالتالي تجنب الحاجة إلى كل من الترجمات المرجعية والمسمى بيانات التدريب. يعتمد المقياس على النهج القائم على أحدث الولاية - وهي مسافة BertScore و Word Mover - من خلال دمج فكرة التشابه الدلالي الحكم. من خلال القيام بذلك، فإنه يحقق ارتباطا أفضل مع درجات بشرية على مجموعات بيانات مختلفة. نظظ أنه يتفوق على هذه المقاييس وغيرها من المقاييس الأخرى في إعداد أحادي الأحادي القياسي (الترجمة المرجعية MT)، كما هو الحال في إعداد ثنائي اللغة المصدر - MT، حيث تنفذ على قدم المساواة مع نهج المربع الزجاجي لتقدير الجودة التي تعتمد على نموذج MT معلومة.
Machine translation (MT) is currently evaluated in one of two ways: in a monolingual fashion, by comparison with the system output to one or more human reference translations, or in a trained crosslingual fashion, by building a supervised model to predict quality scores from human-labeled data. In this paper, we propose a more cost-effective, yet well performing unsupervised alternative SentSim: relying on strong pretrained multilingual word and sentence representations, we directly compare the source with the machine translated sentence, thus avoiding the need for both reference translations and labelled training data. The metric builds on state-of-the-art embedding-based approaches -- namely BERTScore and Word Mover's Distance -- by incorporating a notion of sentence semantic similarity. By doing so, it achieves better correlation with human scores on different datasets. We show that it outperforms these and other metrics in the standard monolingual setting (MT-reference translation), a well as in the source-MT bilingual setting, where it performs on par with glass-box approaches to quality estimation that rely on MT model information.
References used
https://aclanthology.org/
rate research
Read More
Translation quality can be improved by global information from the required target sentence because the decoder can understand both past and future information. However, the model needs additional cost to produce and consider such global information.
In this work, to inject global information but also save cost, we present an efficient method to sample and consider a semantic draft as global information from semantic space for decoding with almost free of cost. Unlike other successful adaptations, we do not have to perform an EM-like process that repeatedly samples a possible semantic from the semantic space. Empirical experiments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. We will open all our source code on GitHub.
Simultaneous machine translation has recently gained traction thanks to significant quality improvements and the advent of streaming applications. Simultaneous translation systems need to find a trade-off between translation quality and response time
, and with this purpose multiple latency measures have been proposed. However, latency evaluations for simultaneous translation are estimated at the sentence level, not taking into account the sequential nature of a streaming scenario. Indeed, these sentence-level latency measures are not well suited for continuous stream translation, resulting in figures that are not coherent with the simultaneous translation policy of the system being assessed. This work proposes a stream level adaptation of the current latency measures based on a re-segmentation approach applied to the output translation, that is successfully evaluated on streaming conditions for a reference IWSLT task.
Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains. It has been recently shown that the probabilities given by a large, multilingual
model can achieve state of the art results when used as a reference-free metric. We experiment with various modifications to this model, and demonstrate that by scaling it up we can match the performance of BLEU. We analyze various potential weaknesses of the approach, and find that it is surprisingly robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities.
Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT models are usually trained on a general domain corpus and then fine-tuned by continuing training on the in-domain corpus.
However, this bears the risk of catastrophic forgetting that the performance on the general domain is decreased drastically. In this work, we propose a new continual learning framework for NMT models. We consider a scenario where the training is comprised of multiple stages and propose a dynamic knowledge distillation technique to alleviate the problem of catastrophic forgetting systematically. We also find that the bias exists in the output linear projection when fine-tuning on the in-domain corpus, and propose a bias-correction module to eliminate the bias. We conduct experiments on three representative settings of NMT application. Experimental results show that the proposed method achieves superior performance compared to baseline models in all settings.
Recent research questions the importance of the dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. In this paper, we push further in this research line and propose a novel substitute
mechanism for self-attention: Recurrent AtteNtion (RAN) . RAN directly learns attention weights without any token-to-token interaction and further improves their capacity by layer-to-layer interaction. Across an extensive set of experiments on 10 machine translation tasks, we find that RAN models are competitive and outperform their Transformer counterpart in certain scenarios, with fewer parameters and inference time. Particularly, when apply RAN to the decoder of Transformer, there brings consistent improvements by about +0.5 BLEU on 6 translation tasks and +1.0 BLEU on Turkish-English translation task. In addition, we conduct extensive analysis on the attention weights of RAN to confirm their reasonableness. Our RAN is a promising alternative to build more effective and efficient NMT models.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
