نلاحظ أن التطوير فقدان انتروبيا فقدان نماذج الترجمة الآلية الخاضعة للإشراف على قوانين الطاقة بمقدار بيانات التدريب وعدد المعلمات غير التضمين في النموذج.نناقش بعض الآثار العملية لهذه النتائج، مثل التنبؤ بلو الذي تحققه نماذج واسعة النطاق وتوقع عائد الاستثمار من بيانات وضع العلامات في أزواج لغة الموارد المنخفضة.
We observe that the development cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.
References used
https://aclanthology.org/
We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactua
Recently, neural machine translation is widely used for its high translation accuracy, but it is also known to show poor performance at long sentence translation. Besides, this tendency appears prominently for low resource languages. We assume that t
Data augmentation, which refers to manipulating the inputs (e.g., adding random noise,masking specific parts) to enlarge the dataset,has been widely adopted in machine learning. Most data augmentation techniques operate on a single input, which limit
Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT models are usually trained on a general domain corpus and then fine-tuned by continuing training on the in-domain corpus.
Recent research questions the importance of the dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. In this paper, we push further in this research line and propose a novel substitute