Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userA Plug-and-Play Method for Controlled Text Generation
طريقة التوصيل والتشغيل للجيل المسيطر
368
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
أظهرت نماذج لغة كبيرة مدربة مسبقا قدرةها مرارا وتكرارا على إنتاج نص يجيد. ومع ذلك حتى عند البدء من موجه، يمكن أن يستمر الجيل في العديد من الاتجاهات المعقولة. طرق فك التشفير الحالية بهدف التحكم في الجيل، على سبيل المثال، لضمان إدراج كلمات محددة، إما أن تتطلب نماذج إضافية أو ضبط جيد، أو العمل بشكل سيء عندما تكون المهمة في متناول اليد، على سبيل المثال، جيل القصة. في هذا العمل، نقدم طريقة فك تشفير التوصيل والتشغيل للتوليد اللغوي السيطرة البسيطة وبديهية، ويمكن وصفها في جملة واحدة: إعطاء موضوع أو كلمة رئيسية، ونضيف التحول إلى توزيع الاحتمالات على المفردات نحو كلمات مماثلة دلالة. نظهر كيف يمكن استخدام صلب هذا التوزيع لفرض قيود صلبة على توليد اللغة، وهو أمر لا تتمكن أي طريقة غيرها من الوسم والتشغيل حاليا مع مولدات لغة SOTA. على الرغم من بساطة هذا النهج، نرى أنه يعمل بشكل جيد بشكل لا يصدق في الممارسة: فك التشفير من GPT-2 يؤدي إلى جمل متنوعة وطلاقة مع ضمان ظهور كلمات دليل معين. نحن نؤدي دراستي المستخدمين، وكشف أن طريقة (1) تتفوقت أساليبنا على الطرق المتنافسة في التقييمات البشرية؛ و (2) إجبار الكلمات الدليلية على الظهور في النص الذي تم إنشاؤه ليس له تأثير على الطلاقة للنص الذي تم إنشاؤه.
Large pre-trained language models have repeatedly shown their ability to produce fluent text. Yet even when starting from a prompt, generation can continue in many plausible directions. Current decoding methods with the goal of controlling generation, e.g., to ensure specific words are included, either require additional models or fine-tuning, or work poorly when the task at hand is semantically unconstrained, e.g., story generation. In this work, we present a plug-and-play decoding method for controlled language generation that is so simple and intuitive, it can be described in a single sentence: given a topic or keyword, we add a shift to the probability distribution over our vocabulary towards semantically similar words. We show how annealing this distribution can be used to impose hard constraints on language generation, something no other plug-and-play method is currently able to do with SOTA language generators. Despite the simplicity of this approach, we see it works incredibly well in practice: decoding from GPT-2 leads to diverse and fluent sentences while guaranteeing the appearance of given guide words. We perform two user studies, revealing that (1) our method outperforms competing methods in human evaluations; and (2) forcing the guide words to appear in the generated text has no impact on the fluency of the generated text.
References used
https://aclanthology.org/
rate research
Read More
Copy mechanisms explicitly obtain unchanged tokens from the source (input) sequence to generate the target (output) sequence under the neural seq2seq framework. However, most of the existing copy mechanisms only consider single word copying from the
source sentences, which results in losing essential tokens while copying long spans. In this work, we propose a plug-and-play architecture, namely BioCopy, to alleviate the problem aforementioned. Specifically, in the training stage, we construct a BIO tag for each token and train the original model with BIO tags jointly. In the inference stage, the model will firstly predict the BIO tag at each time step, then conduct different mask strategies based on the predicted BIO label to diminish the scope of the probability distributions over the vocabulary list. Experimental results on two separate generative tasks show that they all outperform the baseline models by adding our BioCopy to the original model structure.
In the last few years, several methods have been proposed to build meta-embeddings. The general aim was to obtain new representations integrating complementary knowledge from different source pre-trained embeddings thereby improving their overall qua
lity. However, previous meta-embeddings have been evaluated using a variety of methods and datasets, which makes it difficult to draw meaningful conclusions regarding the merits of each approach. In this paper we propose a unified common framework, including both intrinsic and extrinsic tasks, for a fair and objective meta-embeddings evaluation. Furthermore, we present a new method to generate meta-embeddings, outperforming previous work on a large number of intrinsic evaluation benchmarks. Our evaluation framework also allows us to conclude that previous extrinsic evaluations of meta-embeddings have been overestimated.
Existing pre-trained language models (PLMs) have demonstrated the effectiveness of self-supervised learning for a broad range of natural language processing (NLP) tasks. However, most of them are not explicitly aware of domain-specific knowledge, whi
ch is essential for downstream tasks in many domains, such as tasks in e-commerce scenarios. In this paper, we propose K-PLUG, a knowledge-injected pre-trained language model based on the encoder-decoder transformer that can be transferred to both natural language understanding and generation tasks. Specifically, we propose five knowledge-aware self-supervised pre-training objectives to formulate the learning of domain-specific knowledge, including e-commerce domain-specific knowledge-bases, aspects of product entities, categories of product entities, and unique selling propositions of product entities. We verify our method in a diverse range of e-commerce scenarios that require domain-specific knowledge, including product knowledge base completion, abstractive product summarization, and multi-turn dialogue. K-PLUG significantly outperforms baselines across the board, which demonstrates that the proposed method effectively learns a diverse set of domain-specific knowledge for both language understanding and generation tasks. Our code is available.
Large pre-trained neural models have recently shown remarkable progress in text generation. In this paper, we propose to generate text conditioned on the structured data (table) and a prefix (the written text) by leveraging the pre-trained models. We
present a new data-to-text dataset, Table with Written Text (TWT), by repurposing two existing datasets: ToTTo and TabFact. TWT contains both factual and logical statements that are faithful to the structured data, aiming to serve as a useful benchmark for controlled text generation. Compared with existing data-to-text task settings, TWT is more intuitive, the prefix (usually provided by the user) controls the topic of the generated text. Existing methods usually output hallucinated text that is not faithful on TWT. Therefore, we design a novel approach with table-aware attention visibility and copy mechanism over the table. Experimental results show that our approach outperforms state-of-the-art methods under both automatic and human evaluation metrics.
In this paper, we tackle the task of Definition Generation (DG) in Chinese, which aims at automatically generating a definition for a word. Most existing methods take the source word as an indecomposable semantic unit. However, in parataxis languages
like Chinese, word meanings can be composed using the word formation process, where a word (桃花'', peach-blossom) is formed by formation components (桃'', peach; 花'', flower) using a formation rule (Modifier-Head). Inspired by this process, we propose to enhance DG with word formation features. We build a formation-informed dataset, and propose a model DeFT, which Decomposes words into formation features, dynamically Fuses different features through a gating mechanism, and generaTes word definitions. Experimental results show that our method is both effective and robust.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
