يتم وضع تقدير الجودة على مستوى الجملة (QE) من الترجمة الآلية بشكل تقليدي كملقمة الانحدار، ويتم قياس أداء نماذج QE عادة بواسطة ارتباط بيرسون مع ملصقات بشرية. حققت نماذج QE الأخيرة مستويات ارتباطا غير مرئي مسبقا بأحكام بشرية، لكنها تعتمد على نماذج لغوية محلية متعددة اللغات الكبيرة باهظة الثمن بشكل حسابي وجعلها غير ممكنة لتطبيقات العالم الحقيقي. في هذا العمل، نقوم بتقييم العديد من تقنيات ضغط النماذج ل QE والعثور على ذلك، على الرغم من شعبيتها في مهام NLP الأخرى، فإنها تؤدي إلى ضعف الأداء في وضع الانحدار هذا. نلاحظ أن هناك حاجة إلى معلمة نموذجية كاملة لتحقيق نتائج SOTA في مهمة الانحدار. ومع ذلك، فإننا نجادل بأن مستوى التعبير عن نموذج في مجموعة مستمرة غير ضرورية لإحضار تطبيقات المصب في QE، وإظهار أن إعادة صياغة QE كمشكلة تصنيف وتقييم نماذج QE باستخدام مقاييس التصنيف من شأنها أن تعكس أدائها الفعلي بشكل أفضل في الواقع تطبيقات العالم.
Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels. Recent QE models have achieved previously-unseen levels of correlation with human judgments, but they rely on large multilingual contextualized language models that are computationally expensive and make them infeasible for real-world applications. In this work, we evaluate several model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this regression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task. However, we argue that the level of expressiveness of a model in a continuous range is unnecessary given the downstream applications of QE, and show that reframing QE as a classification problem and evaluating QE models using classification metrics would better reflect their actual performance in real-world applications.
References used
https://aclanthology.org/
Services that demanded by users via internet network are classified in two main
kinds, Services work in real time such as video and voice in real time and use UDP
protocol, and other services that work in non-real time such as web browsing (HTTP) a
Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not accou
Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the tas
Quality estimation (QE) of machine translation (MT) aims to evaluate the quality of machine-translated sentences without references and is important in practical applications of MT. Training QE models require massive parallel data with hand-crafted q
We investigate grounded language learning through real-world data, by modelling a teacher-learner dynamics through the natural interactions occurring between users and search engines; in particular, we explore the emergence of semantic generalization