Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userTotal Recall: a Customized Continual Learning Method for Neural Semantic Parsers
إجمالي التذكير: طريقة تعليمية مستمرة مخصصة للمحاورين الدلالي العصبي
57
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تحقق هذه الورقة في التعلم المستمر لتحليل الدلالي. في هذا الإعداد، يتعلم المحلل الدلالي العصبي المهام بالتتابع دون الوصول إلى بيانات التدريب الكامل من المهام السابقة. فشل التطبيق المباشر لخوارزميات التعلم المستمرة لسوتا لهذه المشكلة في تحقيق أداء قابلا للمقارنة مع نماذج إعادة التدريب مع جميع المهام التي شوهدت لأنها لم تعتبر الخصائص الخاصة للنواتج المنظمة التي أسفرت عن المحللين الدلاليين. لذلك، نقترح totalrecall، وهي طريقة تعليمية مستمرة مصممة للمحطات الدلالية العصبية من جوانب: ط) طريقة أخذ العينات لإعادة الذاكرة التي تنويع قوالب النماذج المنطقية وأرصدة توزيعات تصرفات التحليل في الذاكرة؛ 2) طريقة تدريب ذات مرحامة تعمل بشكل كبير على تحسين القدرة على تعميم التعميم من المحللين عبر المهام. نقوم بإجراء تجارب مكثفة لدراسة المشاكل البحثية المتورطة في تحليل دلالي مستمر وتظهر أن المحلل الدلالي العصبي المدرب مع TotalRecall يحقق أداء فائقا من المرء الذي تم تدريبه مباشرة مع خوارزميات التعلم المستمرة SOTA وتحقيق تسريع 3-6 مرات مقارنة بإعادة التدريب من الصفر.
This paper investigates continual learning for semantic parsing. In this setting, a neural semantic parser learns tasks sequentially without accessing full training data from previous tasks. Direct application of the SOTA continual learning algorithms to this problem fails to achieve comparable performance with re-training models with all seen tasks because they have not considered the special properties of structured outputs yielded by semantic parsers. Therefore, we propose TotalRecall, a continual learning method designed for neural semantic parsers from two aspects: i) a sampling method for memory replay that diversifies logical form templates and balances distributions of parse actions in a memory; ii) a two-stage training method that significantly improves generalization capability of the parsers across tasks. We conduct extensive experiments to study the research problems involved in continual semantic parsing and demonstrate that a neural semantic parser trained with TotalRecall achieves superior performance than the one trained directly with the SOTA continual learning algorithms and achieve a 3-6 times speedup compared to re-training from scratch.
References used
https://aclanthology.org/
rate research
Read More
We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Our approach builds on existing losses that encourage attention maps in neural sequence-to-sequence models to imitate the output of cla
ssical word alignment algorithms. Where past work has used word-level alignments, we focus on spans; borrowing ideas from phrase-based machine translation, we align subtrees in semantic parses to spans of input sentences, and encourage neural attention mechanisms to mimic these alignments. This method improves the performance of transformers, RNNs, and structured decoders on three benchmarks of compositional generalization.
In this paper, we propose a globally normalized model for context-free grammar (CFG)-based semantic parsing. Instead of predicting a probability, our model predicts a real-valued score at each step and does not suffer from the label bias problem. Exp
eriments show that our approach outperforms locally normalized models on small datasets, but it does not yield improvement on a large dataset.
Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT models are usually trained on a general domain corpus and then fine-tuned by continuing training on the in-domain corpus.
However, this bears the risk of catastrophic forgetting that the performance on the general domain is decreased drastically. In this work, we propose a new continual learning framework for NMT models. We consider a scenario where the training is comprised of multiple stages and propose a dynamic knowledge distillation technique to alleviate the problem of catastrophic forgetting systematically. We also find that the bias exists in the output linear projection when fine-tuning on the in-domain corpus, and propose a bias-correction module to eliminate the bias. We conduct experiments on three representative settings of NMT application. Experimental results show that the proposed method achieves superior performance compared to baseline models in all settings.
Semantic parsing aims at translating natural language (NL) utterances onto machine-interpretable programs, which can be executed against a real-world environment. The expensive annotation of utterance-program pairs has long been acknowledged as a maj
or bottleneck for the deployment of contemporary neural models to real-life applications. In this work, we focus on the task of semi-supervised learning where a limited amount of annotated data is available together with many unlabeled NL utterances. Based on the observation that programs which correspond to NL utterances should always be executable, we propose to encourage a parser to generate executable programs for unlabeled utterances. Due to the large search space of executable programs, conventional methods that use beam-search for approximation, such as self-training and top-k marginal likelihood training, do not perform as well. Instead, we propose a set of new training objectives that are derived by approaching the problem of learning from executions from the posterior regularization perspective. Our new objectives outperform conventional methods on Overnight and GeoQuery, bridging the gap between semi-supervised and supervised learning.
Synthesizing data for semantic parsing has gained increasing attention recently. However, most methods require handcrafted (high-precision) rules in their generative process, hindering the exploration of diverse unseen data. In this work, we propose
a generative model which features a (non-neural) PCFG that models the composition of programs (e.g., SQL), and a BART-based translation model that maps a program to an utterance. Due to the simplicity of PCFG and pre-trained BART, our generative model can be efficiently learned from existing data at hand. Moreover, explicitly modeling compositions using PCFG leads to better exploration of unseen programs, thus generate more diverse data. We evaluate our method in both in-domain and out-of-domain settings of text-to-SQL parsing on the standard benchmarks of GeoQuery and Spider, respectively. Our empirical results show that the synthesized data generated from our model can substantially help a semantic parser achieve better compositional and domain generalization.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
