Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userGeneralization in Instruction Following Systems
التعميم في التعليمات التالية
163
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
إن فهم وتعليمات اللغة الطبيعية في مجال أساسي هي واحدة من السمات المميزة للذكاء الاصطناعي. في هذه الورقة، نركز على فهم التعليمات في المجال العالمي كتل والتحقيق في قدرات فهم قدرات نظامين أفضل أداء للمهمة. نحن نهدف إلى فهم ما إذا كان أداء اختبار هذه النماذج يشير إلى فهم المجال المكاني وتعليمات اللغة الطبيعية بالنسبة إليها، أو ما إذا كانت مجرد إشارات متفوقة في DataSet. نقوم بصياغة مجموعة من التوقعات قد يكون لدى المرء من التعليمات التالية النموذج وتمييز الأبعاد المختلفة المختلفة التي يجب أن تمتلكها مثل هذا النموذج. على الرغم من أداء الاختبار اللائق، نجد أن النماذج الحديثة تنخفض هذه التوقعات وهشة للغاية. بعد ذلك اقترحنا استراتيجية تعليمية تتضمن تكبير البيانات وإظهارها من خلال تجارب واسعة النطاق التي توليها استراتيجية التعلم المقترحة نماذج تنافسية في مجموعة الاختبار الأصلية مع إرضاء توقعاتنا بشكل أفضل.
Understanding and executing natural language instructions in a grounded domain is one of the hallmarks of artificial intelligence. In this paper, we focus on instruction understanding in the blocks world domain and investigate the language understanding abilities of two top-performing systems for the task. We aim to understand if the test performance of these models indicates an understanding of the spatial domain and of the natural language instructions relative to it, or whether they merely over-fit spurious signals in the dataset. We formulate a set of expectations one might have from an instruction following model and concretely characterize the different dimensions of robustness such a model should possess. Despite decent test performance, we find that state-of-the-art models fall short of these expectations and are extremely brittle. We then propose a learning strategy that involves data augmentation and show through extensive experiments that the proposed learning strategy yields models that are competitive on the original test set while satisfying our expectations much better.
References used
https://aclanthology.org/
rate research
Read More
Executing natural language instructions in a physically grounded domain requires a model that understands both spatial concepts such as left of'' and above'', and the compositional language used to identify landmarks and articulate instructions relat
ive to them. In this paper, we study instruction understanding in the blocks world domain. Given an initial arrangement of blocks and a natural language instruction, the system executes the instruction by manipulating selected blocks. The highly compositional instructions are composed of atomic components and understanding these components is a necessary step to executing the instruction. We show that while end-to-end training (supervised only by the correct block location) fails to address the challenges of this task and performs poorly on instructions involving a single atomic component, knowledge-free auxiliary signals can be used to significantly improve performance by providing supervision for the instruction's components. Specifically, we generate signals that aim at helping the model gradually understand components of the compositional instructions, as well as those that help it better understand spatial concepts, and show their benefit to the overall task for two datasets and two state-of-the-art (SOTA) models, especially when the training data is limited---which is usual in such tasks.
Standard architectures used in instruction following often struggle on novel compositions of subgoals (e.g. navigating to landmarks or picking up objects) observed during training. We propose a modular architecture for following natural language inst
ructions that describe sequences of diverse subgoals. In our approach, subgoal modules each carry out natural language instructions for a specific subgoal type. A sequence of modules to execute is chosen by learning to segment the instructions and predicting a subgoal type for each segment. When compared to standard, non-modular sequence-to-sequence approaches on ALFRED, a challenging instruction following benchmark, we find that modularization improves generalization to novel subgoal compositions, as well as to environments unseen in training.
We analyze language change over time in a collaborative, goal-oriented instructional task, where utility-maximizing participants form conventions and increase their expertise. Prior work studied such scenarios mostly in the context of reference games
, and consistently found that language complexity is reduced along multiple dimensions, such as utterance length, as conventions are formed. In contrast, we find that, given the ability to increase instruction utility, instructors increase language complexity along these previously studied dimensions to better collaborate with increasingly skilled instruction followers.
Although neural sequence-to-sequence models have been successfully applied to semantic parsing, they fail at compositional generalization, i.e., they are unable to systematically generalize to unseen compositions of seen components. Motivated by trad
itional semantic parsing where compositionality is explicitly accounted for by symbolic grammars, we propose a new decoding framework that preserves the expressivity and generality of sequence-to-sequence models while featuring lexicon-style alignments and disentangled information processing. Specifically, we decompose decoding into two phases where an input utterance is first tagged with semantic symbols representing the meaning of individual words, and then a sequence-to-sequence model is used to predict the final meaning representation conditioning on the utterance and the predicted tag sequence. Experimental results on three semantic parsing datasets show that the proposed approach consistently improves compositional generalization across model architectures, domains, and semantic formalisms.
Much of recent progress in NLU was shown to be due to models' learning dataset-specific heuristics. We conduct a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (ada
pters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. We report 2 successful and 3 unsuccessful strategies, all providing insights into how Transformer-based models learn to generalize.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
