أدت نماذج اللغة مثل GPT-2 بشكل جيد على إنشاء جمل سليمة نصنية لمهام إكمال تلقائي للنصوص.ومع ذلك، غالبا ما تتطلب هذه النماذج جهدا تدريبيا كبيرا للتكيف مع مجالات الكتابة المحددة (على سبيل المثال، الطبية).في هذه الورقة، نقترح استراتيجية تدريبية متوسطة لتعزيز أداء نماذج اللغة المدربة مسبقا في مهمة إكمال تلقائي النص وتكييفها بشكل مستقل إلى مجالات محددة.تضم استراتيجيتنا هدفا جديدا للتدريب على الإشراف على الذات يسمى التنبؤ بالعبارة التالية (NPP)، والذي يشجع نموذج اللغة لإكمال الاستعلام الجزئي مع العبارات المخصبة وتحسين أداء الانتهاء التلقائي للنموذج في النهاية.أظهرت التجارب الأولية أن نهجنا قادر على تفوق خطوط الأساس في الإنجاز التلقائي للنطاقات البريدية والكتابة الأكاديمية.
Language models such as GPT-2 have performed well on constructing syntactically sound sentences for text auto-completion tasks. However, such models often require considerable training effort to adapt to specific writing domains (e.g., medical). In this paper, we propose an intermediate training strategy to enhance pre-trained language models' performance in the text auto-completion task and fastly adapt them to specific domains. Our strategy includes a novel self-supervised training objective called Next Phrase Prediction (NPP), which encourages a language model to complete the partial query with enriched phrases and eventually improve the model's text auto-completion performance. Preliminary experiments have shown that our approach is able to outperform the baselines in auto-completion for email and academic-writing domains.
References used
https://aclanthology.org/
The quality of fully automated text simplification systems is not good enough for use in real-world settings; instead, human simplifications are used. In this paper, we examine how to improve the cost and quality of human simplifications by leveragin
Graph convolutional networks (GCNs) have been applied recently to text classification and produced an excellent performance. However, existing GCN-based methods do not assume an explicit latent semantic structure of documents, making learned represen
Point-of-interest (POI) type prediction is the task of inferring the type of a place from where a social media post was shared. Inferring a POI's type is useful for studies in computational social science including sociolinguistics, geosemiotics, and
Recent studies have shown that a bias in thetext suggestions system can percolate in theuser's writing. In this pilot study, we ask thequestion: How do people interact with text pre-diction models, in an inline next phrase sugges-tion interface and h
We describe our submissions to the 6th edition of the Social Media Mining for Health Applications (SMM4H) shared task. Our team (OGNLP) participated in the sub-task: Classification of tweets self-reporting potential cases of COVID-19 (Task 5). For ou