Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new usercushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE
Cushlepor: تخصيص متري هلكارد باستخدام Optuna لاتفاق أعلى مع الأحكام البشرية أو Lange Language Model
105
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
كان التقييم البشري مكلفا دائما بينما يكافح الباحثون من أجل الثقة في المقاييس التلقائية. لمعالجة هذا، نقترح تخصيص المقاييس التقليدية من خلال أخذ مزايا نماذج اللغة المدربة مسبقا (PLMS) وعشرات المحدودة المسمى الإنسان المسمى. نقدم أولا عوامل HLEPOR متري، تليها نسخة بيثون التي طورناها (استدارها) التي حققت الضبط التلقائي لمعلمات الترجيح في هلكور متري. ثم نقدم HLEPOR (Cushlepor) المخصص الذي يستخدم إطار عمل تحسين المعلمات Optuna إلى معلمات HLEOR Tune-Tune Tune نحو اتفاق أفضل لاتفاق أفضل على نماذج اللغة المدربة مسبقا (باستخدام LAST) فيما يتعلق بأزواج لغة MT الدقيقة التي يتم نشرها. نحن أيضا تحسين cushlepor تجاه بيانات التقييم البشرية المهنية بناء على إطار إدارة الموارد البشرية و PSQM على أزواج اللغة الإنجليزية والألمانية والإنجليزية. تظهر التحقيقات التجريبية Cushlepor يعزز عروض هيلور بورز نحو اتفاقات أفضل للمتمرنات مثل LAST مع تكلفة أقل بكثير، واتفاقيات أفضل للتقييمات البشرية بما في ذلك درجات MQM و PSQM، وتوجز أداء أفضل بكثير من بلو. تظهر النتائج الرسمية أن عروضنا تفوز بثلاث أزواج لغوية بما في ذلك اللغة الإنجليزية والألمانية والصينية الإنجليزية على نطاق الأخبار عبر Cushlepor (LM) والإنجليزية-الروسية على نطاق TED عبر HLEPOR. (البيانات المتاحة في https://github.com/poethan/cushlepor)
Human evaluation has always been expensive while researchers struggle to trust the automatic metrics. To address this, we propose to customise traditional metrics by taking advantages of the pre-trained language models (PLMs) and the limited available human labelled scores. We first re-introduce the hLEPOR metric factors, followed by the Python version we developed (ported) which achieved the automatic tuning of the weighting parameters in hLEPOR metric. Then we present the customised hLEPOR (cushLEPOR) which uses Optuna hyper-parameter optimisation framework to fine-tune hLEPOR weighting parameters towards better agreement to pre-trained language models (using LaBSE) regarding the exact MT language pairs that cushLEPOR is deployed to. We also optimise cushLEPOR towards professional human evaluation data based on MQM and pSQM framework on English-German and Chinese-English language pairs. The experimental investigations show cushLEPOR boosts hLEPOR performances towards better agreements to PLMs like LABSE with much lower cost, and better agreements to human evaluations including MQM and pSQM scores, and yields much better performances than BLEU. Official results show that our submissions win three language pairs including English-German and Chinese-English on News domain via cushLEPOR(LM) and English-Russian on TED domain via hLEPOR. (data available at https://github.com/poethan/cushLEPOR)
References used
https://aclanthology.org/
rate research
Read More
Commit message is a document that summarizes source code changes in natural language. A good commit message clearly shows the source code changes, so this enhances collaboration between developers. Therefore, our work is to develop a model that autom
atically writes the commit message. To this end, we release 345K datasets consisting of code modification and commit messages in six programming languages (Python, PHP, Go, Java, JavaScript, and Ruby). Similar to the neural machine translation (NMT) model, using our dataset, we feed the code modification to the encoder input and the commit message to the decoder input and measure the result of the generated commit message with BLEU-4. Also, we propose the following two training methods to improve the result of generating the commit message: (1) A method of preprocessing the input to feed the code modification to the encoder input. (2) A method that uses an initial weight suitable for the code domain to reduce the gap in contextual representation between programming language (PL) and natural language (NL).
Recent work has demonstrated that pre-training in-domain language models can boost performance when adapting to a new domain. However, the costs associated with pre-training raise an important question: given a fixed budget, what steps should an NLP
practitioner take to maximize performance? In this paper, we study domain adaptation under budget constraints, and approach it as a customer choice problem between data annotation and pre-training. Specifically, we measure the annotation cost of three procedural text datasets and the pre-training cost of three in-domain language models. Then we evaluate the utility of different combinations of pre-training and data annotation under varying budget constraints to assess which combination strategy works best. We find that, for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, a combination of data annotation and in-domain pre-training works more optimally. We therefore suggest that task-specific data annotation should be part of an economical strategy when adapting an NLP model to a new domain.
Compared to monolingual models, cross-lingual models usually require a more expressive vocabulary to represent all languages adequately. We find that many languages are under-represented in recent cross-lingual language models due to the limited voca
bulary capacity. To this end, we propose an algorithm VoCap to determine the desired vocabulary capacity of each language. However, increasing the vocabulary size significantly slows down the pre-training speed. In order to address the issues, we propose k-NN-based target sampling to accelerate the expensive softmax. Our experiments show that the multilingual vocabulary learned with VoCap benefits cross-lingual language model pre-training. Moreover, k-NN-based target sampling mitigates the side-effects of increasing the vocabulary size while achieving comparable performance and faster pre-training speed. The code and the pretrained multilingual vocabularies are available at https://github.com/bozheng-hit/VoCapXLM.
Fine-tuned pre-trained language models (LMs) have achieved enormous success in many natural language processing (NLP) tasks, but they still require excessive labeled data in the fine-tuning stage. We study the problem of fine-tuning pre-trained LMs u
sing only weak supervision, without any labeled data. This problem is challenging because the high capacity of LMs makes them prone to overfitting the noisy labels generated by weak supervision. To address this problem, we develop a contrastive self-training framework, COSINE, to enable fine-tuning LMs with weak supervision. Underpinned by contrastive regularization and confidence-based reweighting, our framework gradually improves model fitting while effectively suppressing error propagation. Experiments on sequence, token, and sentence pair classification tasks show that our model outperforms the strongest baseline by large margins and achieves competitive performance with fully-supervised fine-tuning methods. Our implementation is available on https://github.com/yueyu1030/COSINE.
Metaphors are ubiquitous in natural language, and detecting them requires contextual reasoning about whether a semantic incongruence actually exists. Most existing work addresses this problem using pre-trained contextualized models. Despite their suc
cess, these models require a large amount of labeled data and are not linguistically-based. In this paper, we proposed a ContrAstive pre-Trained modEl (CATE) for metaphor detection with semi-supervised learning. Our model first uses a pre-trained model to obtain a contextual representation of target words and employs a contrastive objective to promote an increased distance between target words' literal and metaphorical senses based on linguistic theories. Furthermore, we propose a simple strategy to collect large-scale candidate instances from the general corpus and generalize the model via self-training. Extensive experiments show that CATE achieves better performance against state-of-the-art baselines on several benchmark datasets.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
