تم التركيز الفكرة الرئيسية لهذا الحل على التركيز على تنظيف Corpus وإعدادها وبعد ذلك، استخدم حل خارج مربع (OpenNMT) مع طراز المحولات المنشور الافتراضي.لإعداد Corpus، استخدمنا مجموعة من الأدوات القياسية (كبرامج نصية موسى أو حزم بيثون)، ولكن أيضا، من بين البرامج النصية الثابتة الأخرى، ومزخرفة مخصصة لبثون مع القدرة على استبدال الأرقام للمتغيرات، حل مشكلة العلوي / السفلي للحالةالمفردات وتوفير تجزئة جيدة لمعظم علامات الترقيم.لقد بدأنا أيضا خطا لتنظيف Corpus بناء على تقدير الاحتمالات الإحصائية ل Corpus المستهدف المصدر، مع نتائج غير واضحة.أيضا، لقد قمت بتشغيل بعض الاختبارات مع تجزئة الكلمات المقاطعة، مرة أخرى بنتائج غير واضحة، لذلك في النهاية، بعد تكتيح جملة Word، استخدمنا جملة BPE الخاصة بوحدات الكلمات الفرعية لتغذية OpenNMT.
The main idea of this solution has been to focus on corpus cleaning and preparation and after that, use an out of box solution (OpenNMT) with its default published transformer model. To prepare the corpus, we have used set of standard tools (as Moses scripts or python packages), but also, among other python scripts, a python custom tokenizer with the ability to replace numbers for variables, solve the upper/lower case issue of the vocabulary and provide good segmentation for most of the punctuation. We also have started a line to clean corpus based on statistical probability estimation of source-target corpus, with unclear results. Also, we have run some tests with syllabical word segmentation, again with unclear results, so at the end, after word sentence tokenization we have used BPE SentencePiece for subword units to feed OpenNMT.
References used
https://aclanthology.org/
This paper describes the SEBAMAT contribution to the 2021 WMT Similar Language Translation shared task. Using the Marian neural machine translation toolkit, translation systems based on Google's transformer architecture were built in both directions
Offensive language detection and analysis has become a major area of research in Natural Language Processing. The freedom of participation in social media has exposed online users to posts designed to denigrate, insult or hurt them according to gende
In this work, two Neural Machine Translation (NMT) systems have been developed and evaluated as part of the bidirectional Tamil-Telugu similar languages translation subtask in WMT21. The OpenNMT-py toolkit has been used to create quick prototypes of
We investigate transfer learning based on pre-trained neural machine translation models to translate between (low-resource) similar languages. This work is part of our contribution to the WMT 2021 Similar Languages Translation Shared Task where we su
This paper is concerned with the theme of divine
justice in The Spanish Tragedy (1592) by Thomas Kyd
(1558-1594), a famous English Elizabethan dramatist. It
first defines the term "didactic", and then moves to
discuss the Portuguese playlet in Th