Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userShould we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification
هل يجب أن نجد نموذجا آخر؟: تحسين أداء ترجمة الآلات العصبية مع طريقة تكييف قطعة واحدة دون تعديل نموذجي
727
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تعتمد معظم دراسات معالجة اللغة الطبيعية الأخيرة (NLP) على النهج الضميني - PFA (PFA)، ولكن في المؤسسات أو الشركات الصغيرة والمتوسطة مع الأجهزة غير الكافية، هناك العديد من القيود لخدمة برنامج تطبيق NLP باستخدام هذه التكنولوجيا بسبب هذه التكنولوجيا سرعة بطيئة وذاكرة غير كافية. تتطلب أحدث تقنيات PFA كميات كبيرة من البيانات، خاصة لغات الموارد المنخفضة، مما يجعلها أكثر صعوبة في العمل معها. نقترح طريقة جديدة للتكامل، قطعة واحدة، لمعالجة هذا القيد الذي يجمع بين طريقة تشغيل الكلمات الفرعية التي تعتبر الكلمات الفرعية التي تعتبرها المورفولوجيا وطريقة المفردات المستخدمة بعد التحقيق في طريقة حالية لم تعتبر بعناية من قبل. يمكن أيضا استخدام طريقةنا المقترحة دون تعديل هيكل النموذج. نقوم بتجربة تطبيق قطعة واحدة إلى اللغة الكورية والغنية بالموراطية والموارد المنخفضة. ونحن نستمد نتيجة تكييف الكلمات الفرعية المثلى للترجمة الآلية الكورية والإنجليزية من خلال إجراء دراسة حالة تجمع بين طريقة تكتيح الكلمات الفرعية، والتجزئة المورفولوجية، وطريقة المفردات. من خلال التجارب المقارنة مع جميع أساليب التكامل المستخدمة حاليا في بحث NLP، تحقق قطعة واحدة أداء قابلة للمقارنة مع النموذج الحالي للترجمة الآلية الكورية والإنجليزية الحالية.
Most of the recent Natural Language Processing(NLP) studies are based on the Pretrain-Finetuning Approach (PFA), but in small and medium-sized enterprises or companies with insufficient hardware there are many limitations to servicing NLP application software using such technology due to slow speed and insufficient memory. The latest PFA technologies require large amounts of data, especially for low-resource languages, making them much more difficult to work with. We propose a new tokenization method, ONE-Piece, to address this limitation that combines the morphology-considered subword tokenization method and the vocabulary method used after probing for an existing method that has not been carefully considered before. Our proposed method can also be used without modifying the model structure. We experiment by applying ONE-Piece to Korean, a morphologically-rich and low-resource language. We derive an optimal subword tokenization result for Korean-English machine translation by conducting a case study that combines the subword tokenization method, morphological segmentation, and vocabulary method. Through comparative experiments with all the tokenization methods currently used in NLP research, ONE-Piece achieves performance comparable to the current Korean-English machine translation state-of-the-art model.
References used
https://aclanthology.org/
rate research
Read More
Multilingual neural machine translation models typically handle one source language at a time. However, prior work has shown that translating from multiple source languages improves translation quality. Different from existing approaches on multi-sou
rce translation that are limited to the test scenario where parallel source sentences from multiple languages are available at inference time, we propose to improve multilingual translation in a more common scenario by exploiting synthetic source sentences from auxiliary languages. We train our model on synthetic multi-source corpora and apply random masking to enable flexible inference with single-source or bi-source inputs. Extensive experiments on Chinese/English-Japanese and a large-scale multilingual translation benchmark show that our model outperforms the multilingual baseline significantly by up to +4.0 BLEU with the largest improvements on low-resource or distant language pairs.
Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via UTF-8,
obviating the need for an embedding layer since there are fewer token types (256) than dimensions. Surprisingly, replacing the ubiquitous embedding layer with one-hot representations of each byte does not hurt performance; experiments on byte-to-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models. A deeper investigation reveals that the combination of embeddingless models with decoder-input dropout amounts to token dropout, which benefits byte-to-byte models in particular.
Multilingual Neural Machine Translation (MNMT) trains a single NMT model that supports translation between multiple languages, rather than training separate models for different languages. Learning a single model can enhance the low-resource translat
ion by leveraging data from multiple languages. However, the performance of an MNMT model is highly dependent on the type of languages used in training, as transferring knowledge from a diverse set of languages degrades the translation performance due to negative transfer. In this paper, we propose a Hierarchical Knowledge Distillation (HKD) approach for MNMT which capitalises on language groups generated according to typological features and phylogeny of languages to overcome the issue of negative transfer. HKD generates a set of multilingual teacher-assistant models via a selective knowledge distillation mechanism based on the language groups, and then distills the ultimate multilingual model from those assistants in an adaptive way. Experimental results derived from the TED dataset with 53 languages demonstrate the effectiveness of our approach in avoiding the negative transfer effect in MNMT, leading to an improved translation performance (about 1 BLEU score in average) compared to strong baselines.
Most work in NLP makes the assumption that it is desirable to develop solutions in the native language in question. There is consequently a strong trend towards building native language models even for low-resource languages. This paper questions thi
s development, and explores the idea of simply translating the data into English, thereby enabling the use of pretrained, and large-scale, English language models. We demonstrate empirically that a large English language model coupled with modern machine translation outperforms native language models in most Scandinavian languages. The exception to this is Finnish, which we assume is due to inferior translation quality. Our results suggest that machine translation is a mature technology, which raises a serious counter-argument for training native language models for low-resource languages. This paper therefore strives to make a provocative but important point. As English language models are improving at an unprecedented pace, which in turn improves machine translation, it is from an empirical and environmental stand-point more effective to translate data from low-resource languages into English, than to build language models for such languages.
We participated in all tracks of the WMT 2021 efficient machine translation task: single-core CPU, multi-core CPU, and GPU hardware with throughput and latency conditions. Our submissions combine several efficiency strategies: knowledge distillation,
a simpler simple recurrent unit (SSRU) decoder with one or two layers, lexical shortlists, smaller numerical formats, and pruning. For the CPU track, we used quantized 8-bit models. For the GPU track, we experimented with FP16 and 8-bit integers in tensorcores. Some of our submissions optimize for size via 4-bit log quantization and omitting a lexical shortlist. We have extended pruning to more parts of the network, emphasizing component- and block-level pruning that actually improves speed unlike coefficient-wise pruning.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
