Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userNews Headline Grouping as a Challenging NLU Task
عنوان الأخبار تجميع مهمة NLU صعبة
138
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
لقد شهد التقدم الأخير في فهم اللغة الطبيعية (NLU) أحدث النماذج تفوق الأداء البشري على العديد من المهام القياسية. قادت هذه النتائج المثيرة للإعجاب المجتمع إلى التقاطية بشأن قيود البيانات، وتكرر التحديات الأكثر دقة. في هذه الورقة، نقدم مهمة تجميع العنوان الرئيسي (HLG) ومجموعة بيانات مقابلة (HLGD) تتكون من 20،056 أزواج من عناوين الأخبار، والتي تم تسمية كل منها بحكم ثنائي فيما إذا كان الزوج ينتمي إلى نفس المجموعة. على HLGD، يحقق المعلقون البشري الأداء العالي حوالي 0.9 F-1، في حين تصل نماذج المحولات الحالية من المحولات الحالية إلى 0.75 F-1، وفتح المسار لمزيد من التحسينات. نقترحنا كذلك نموذج مبادلة عناوين رئيسية رواية غير مدهشة لمهمة تجميع العنوان الرئيسي الذي يحقق في غضون 3 F-1 من أفضل النموذج الإشرافي. أخيرا، نقوم بتحليل نماذج عالية الأداء مع اختبارات الاتساق، وتجد أن النماذج ليست متسقة في تنبؤاتها، وكشف عن حدود النمذجة من الهندسة الحالية.
Recent progress in Natural Language Understanding (NLU) has seen the latest models outperform human performance on many standard tasks. These impressive results have led the community to introspect on dataset limitations, and iterate on more nuanced challenges. In this paper, we introduce the task of HeadLine Grouping (HLG) and a corresponding dataset (HLGD) consisting of 20,056 pairs of news headlines, each labeled with a binary judgement as to whether the pair belongs within the same group. On HLGD, human annotators achieve high performance of around 0.9 F-1, while current state-of-the art Transformer models only reach 0.75 F-1, opening the path for further improvements. We further propose a novel unsupervised Headline Generator Swap model for the task of HeadLine Grouping that achieves within 3 F-1 of the best supervised model. Finally, we analyze high-performing models with consistency tests, and find that models are not consistent in their predictions, revealing modeling limits of current architectures.
References used
https://aclanthology.org/
rate research
Read More
We submitted two uni-directional models, one for English→Icelandic direction and other for Icelandic→English direction. Our news translation system is based on the transformer-big architecture, it makes use of corpora filtering, back-translation and
forward translation applied to parallel and monolingual data alike
Deep Learning-based NLP systems can be sensitive to unseen tokens and hard to learn with high-dimensional inputs, which critically hinder learning generalization. We introduce an approach by grouping input words based on their semantic diversity to s
implify input language representation with low ambiguity. Since the semantically diverse words reside in different contexts, we are able to substitute words with their groups and still distinguish word meanings relying on their contexts. We design several algorithms that compute diverse groupings based on random sampling, geometric distances, and entropy maximization, and we prove formal guarantees for the entropy-based algorithms. Experimental results show that our methods generalize NLP models and demonstrate enhanced accuracy on POS tagging and LM tasks and significant improvements on medium-scale machine translation tasks, up to +6.5 BLEU points. Our source code is available at https://github.com/abdulrafae/dg.
For any E-commerce website it is a nontrivial problem to build enduring advertisements that attract shoppers. It is hard to pass the creative quality bar of the website, especially at a large scale. We thus propose a programmatic solution to generate
product advertising headlines using retail content. We propose a state of the art application of Reinforcement Learning (RL) Policy gradient methods on Transformer (Vaswani et al., 2017) based Masked Language Models (Devlin et al., 2019). Our method creates the advertising headline by jointly conditioning on multiple products that a seller wishes to advertise. We demonstrate that our method outperforms existing Transformer and LSTM + RL methods in overlap metrics and quality audits. We also show that our model generated headlines outperform human submitted headlines in terms of both grammar and creative quality as determined by audits.
Natural Language Processing (NLP) is defined by specific, separate tasks, with each their own literature, benchmark datasets, and definitions. In this position paper, we argue that for a complex problem such as the threat to democracy by non-diverse
news recommender systems, it is important to take into account a higher-order, normative goal and its implications. Experts in ethics, political science and media studies have suggested that news recommendation systems could be used to support a deliberative democracy. We reflect on the role of NLP in recommendation systems with this specific goal in mind and show that this theory of democracy helps to identify which NLP tasks and techniques can support this goal, and what work still needs to be done. This leads to recommendations for NLP researchers working on this specific problem as well as researchers working on other complex multidisciplinary problems.
Many crowdsourced NLP datasets contain systematic artifacts that are identified only after data collection is complete. Earlier identification of these issues should make it easier to create high-quality training and evaluation data. We attempt this
by evaluating protocols in which expert linguists work in the loop' during data collection to identify and address these issues by adjusting task instructions and incentives. Using natural language inference as a test case, we compare three data collection protocols: (i) a baseline protocol with no linguist involvement, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints on the writing task, and (iii) an extension that adds direct interaction between linguists and crowdworkers via a chatroom. We find that linguist involvement does not lead to increased accuracy on out-of-domain test sets compared to baseline, and adding a chatroom has no effect on the data. Linguist involvement does, however, lead to more challenging evaluation data and higher accuracy on some challenge sets, demonstrating the benefits of integrating expert analysis during data collection.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
