Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userBuilding Adaptive Acceptability Classifiers for Neural NLG
بناء منصوص مقبولية التكيف ل NLG العصبي
278
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نقترح إطارا جديدا لتدريب النماذج لتصنيف مقبولية الردود الناتجة عن نماذج توليد اللغة الطبيعية (NLG)، وتحسين تحويل الجملة الحالية والنهج القائمة على النماذج. يعتبر استجابة NLG مقبولة إذا كانت كل من صحيحة وتجميعها. نحن لا نستخدم أي مراجع بشرية مما يجعل المصنفين مناسبين لنشر وقت التشغيل. يتم الحصول على بيانات التدريب للصفوف المصنفة باستخدام نهج من 2 مراحل من أول توليد البيانات الاصطناعية باستخدام مزيج من النهج القائمة والجديدة القائمة على النموذج متبوعا بإطار التحقق من صحة جديدة لتصفية وفرز البيانات الاصطناعية في فئات مقبولة وغير مقبولة. يتكيف نهجنا ذو المرحلتين لدينا بمجموعة واسعة من تمثيل البيانات ولا يتطلب بيانات إضافية تتجاوز ما يتم تدريب نماذج NLG عليها. وهي أيضا مستقلة عن الهندسة المعمارية النموذجية NLG الأساسية، وقادرة على توليد عينات أكثر واقعية قريبة من توزيع الردود الناتجة عن النموذج NLG. نقدم النتائج على 5 مجموعات بيانات (Webnlg، وتنظيفها E2E، Viggo، والتنبيه، والطقس) بتمثيلات بيانات متفاوتة. قارن إطار عملنا مع التقنيات الحالية التي تنطوي على توليد بيانات الاصطناعية باستخدام تحويلات جملة بسيطة و / أو تقنيات قائمة على الطراز، وإظهار أن بناء مصنف مقبولية يستخدمون البيانات التي تشبه مخرجات طراز الجيل تتبع إطار التحقق من الصحة تتفوق على التقنيات الحالية، وتحقيق الدولة النتائج من الفن. نظهر أيضا أن تقنياتنا يمكن استخدامها في إعدادات قليلة عند استخدام التدريب الذاتي.
We propose a novel framework to train models to classify acceptability of responses generated by natural language generation (NLG) models, improving upon existing sentence transformation and model-based approaches. An NLG response is considered acceptable if it is both semantically correct and grammatical. We don't make use of any human references making the classifiers suitable for runtime deployment. Training data for the classifiers is obtained using a 2-stage approach of first generating synthetic data using a combination of existing and new model-based approaches followed by a novel validation framework to filter and sort the synthetic data into acceptable and unacceptable classes. Our 2-stage approach adapts to a wide range of data representations and does not require additional data beyond what the NLG models are trained on. It is also independent of the underlying NLG model architecture, and is able to generate more realistic samples close to the distribution of the NLG model-generated responses. We present results on 5 datasets (WebNLG, Cleaned E2E, ViGGO, Alarm, and Weather) with varying data representations. We compare our framework with existing techniques that involve synthetic data generation using simple sentence transformations and/or model-based techniques, and show that building acceptability classifiers using data that resembles the generation model outputs followed by a validation framework outperforms the existing techniques, achieving state-of-the-art results. We also show that our techniques can be used in few-shot settings using self-training.
References used
https://aclanthology.org/
rate research
Read More
Neural approaches to natural language generation in task-oriented dialogue have typically required large amounts of annotated training data to achieve satisfactory performance, especially when generating from compositional inputs. To address this iss
ue, we show that self-training enhanced with constrained decoding yields large gains in data efficiency on a conversational weather dataset that employs compositional meaning representations. In particular, our experiments indicate that self-training with constrained decoding can enable sequence-to-sequence models to achieve satisfactory quality using vanilla decoding with five to ten times less data than with ordinary supervised baseline; moreover, by leveraging pretrained models, data efficiency can be increased further to fifty times. We confirm the main automatic results with human evaluations and show that they extend to an enhanced, compositional version of the E2E dataset. The end result is an approach that makes it possible to achieve acceptable performance on compositional NLG tasks using hundreds rather than tens of thousands of training samples.
We introduce SelfExplain, a novel self-explaining model that explains a text classifier's predictions using phrase-based concepts. SelfExplain augments existing neural classifiers by adding (1) a globally interpretable layer that identifies the most
influential concepts in the training set for a given sample and (2) a locally interpretable layer that quantifies the contribution of each local input concept by computing a relevance score relative to the predicted label. Experiments across five text-classification datasets show that SelfExplain facilitates interpretability without sacrificing performance. Most importantly, explanations from SelfExplain show sufficiency for model predictions and are perceived as adequate, trustworthy and understandable by human judges compared to existing widely-used baselines.
We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-base
d architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.
In this paper, we present the first statistical parser for Lambek categorial grammar (LCG), a grammatical formalism for which the graphical proof method known as *proof nets* is applicable. Our parser incorporates proof net structure and constraints
into a system based on self-attention networks via novel model elements. Our experiments on an English LCG corpus show that incorporating term graph structure is helpful to the model, improving both parsing accuracy and coverage. Moreover, we derive novel loss functions by expressing proof net constraints as differentiable functions of our model output, enabling us to train our parser without ground-truth derivations.
Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we obser
ve that the human evaluation scores on these multiple criteria are often not correlated. For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores assigned by humans for a single criteria (overall quality) alone is inadequate. Indeed, our extensive study involving 25 automatic evaluation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with human scores on all desirable criteria, for most NLG tasks. Given this situation, we propose CheckLists for better design and evaluation of automatic metrics. We design templates which target a specific criteria (e.g., coverage) and perturb the output such that the quality gets affected only along this specific criteria (e.g., the coverage drops). We show that existing evaluation metrics are not robust against even such simple perturbations and disagree with scores assigned by humans to the perturbed output. The proposed templates thus allow for a fine-grained assessment of automatic evaluation metrics exposing their limitations and will facilitate better design, analysis and evaluation of such metrics. Our templates and code are available at https://iitmnlp.github.io/EvalEval/
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
