Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userEmpathetic BERT2BERT Conversational Model: Learning Arabic Language Generation with Little Data
التعاطفية Bert2bert نموذج المحادثة: تعلم توليد اللغة العربية مع القليل من البيانات
609
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تمكين السلوك التعرفي في وكلاء الحوار باللغة العربية هو جانب مهم في بناء نماذج المحادثة يشبه الإنسان. في حين أن معالجة اللغة العربية الطبيعية قد شهدت تطورات كبيرة في فهم اللغة الطبيعية (NLU) مع نماذج اللغة مثل أرابيرت، فإن توليد اللغة الطبيعية (NLG) لا تزال تحديا. تعد أوجه القصور النموذجية لنماذج فك تشفير NLG في المقام الأول إلى عدم وجود مجموعات البيانات العربية مناسبة لتدريب نماذج NLG مثل عوامل المحادثة. للتغلب على هذه المسألة، نقترح فك ترميز التركيب المستندة إلى المحولات مع معلمات أرابتير. من خلال تهيئة أوزان التشفير والكشف عن الأوزان المدربة مسبقا مسبقا، كان طرازنا قادرا على الاستفادة من نقل المعرفة وزيادة الأداء في توليد الاستجابة. لتمكين التعاطف في نموذج المحادثة لدينا، نربطها باستخدام مجموعة بيانات ArabithatheticTialogues وتحقيق الأداء العالي في توليد الاستجابة المتعاطفة. على وجه التحديد، حقق نموذجنا قيمة حيرة منخفضة تتراوح بين 17.0 وزيادة في 5 نقاط بلو مقارنة بالنموذج السابق للدولة السابقة. أيضا، تم تصنيف نموذجنا المقترح بشدة بنسبة 85 مقيم بشري، والتحقق من قادرته عالية في إظهار التعاطف مع توليد الاستجابات ذات الصلة والطلاقة في إعدادات المجال المفتوح.
Enabling empathetic behavior in Arabic dialogue agents is an important aspect of building human-like conversational models. While Arabic Natural Language Processing has seen significant advances in Natural Language Understanding (NLU) with language models such as AraBERT, Natural Language Generation (NLG) remains a challenge. The shortcomings of NLG encoder-decoder models are primarily due to the lack of Arabic datasets suitable to train NLG models such as conversational agents. To overcome this issue, we propose a transformer-based encoder-decoder initialized with AraBERT parameters. By initializing the weights of the encoder and decoder with AraBERT pre-trained weights, our model was able to leverage knowledge transfer and boost performance in response generation. To enable empathy in our conversational model, we train it using the ArabicEmpatheticDialogues dataset and achieve high performance in empathetic response generation. Specifically, our model achieved a low perplexity value of 17.0 and an increase in 5 BLEU points compared to the previous state-of-the-art model. Also, our proposed model was rated highly by 85 human evaluators, validating its high capability in exhibiting empathy while generating relevant and fluent responses in open-domain settings.
References used
https://aclanthology.org/
rate research
Read More
We analyse how a transformer-based language model learns the rules of chess from text data of recorded games. We show how it is possible to investigate how the model capacity and the available number of training data influence the learning success of
a language model with the help of chess-specific metrics. With these metrics, we show that more games used for training in the studied range offers significantly better results for the same training time. However, model size does not show such a clear influence. It is also interesting to observe that the usual evaluation metrics for language models, predictive accuracy and perplexity, give no indication of this here. Further examination of trained models reveals how they store information about board state in the activations of neuron groups, and how the overall sequence of previous moves influences the newly-generated moves.
Despite its proven efficiency in other fields, data augmentation is less popular in the context of natural language processing (NLP) due to its complexity and limited results. A recent study (Longpre et al., 2020) showed for example that task-agnosti
c data augmentations fail to consistently boost the performance of pretrained transformers even in low data regimes. In this paper, we investigate whether data-driven augmentation scheduling and the integration of a wider set of transformations can lead to improved performance where fixed and limited policies were unsuccessful. Our results suggest that, while this approach can help the training process in some settings, the improvements are unsubstantial. This negative result is meant to help researchers better understand the limitations of data augmentation for NLP.
Abstract Recent approaches to data-to-text generation have adopted the very successful encoder-decoder architecture or variants thereof. These models generate text that is fluent (but often imprecise) and perform quite poorly at selecting appropriate
content and ordering it coherently. To overcome some of these issues, we propose a neural model with a macro planning stage followed by a generation stage reminiscent of traditional methods which embrace separate modules for planning and surface realization. Macro plans represent high level organization of important content such as entities, events, and their interactions; they are learned from data and given as input to the generator. Extensive experiments on two data-to-text benchmarks (RotoWire and MLB) show that our approach outperforms competitive baselines in terms of automatic and human evaluation.
Current approaches to empathetic response generation focus on learning a model to predict an emotion label and generate a response based on this label and have achieved promising results. However, the emotion cause, an essential factor for empathetic
responding, is ignored. The emotion cause is a stimulus for human emotions. Recognizing the emotion cause is helpful to better understand human emotions so as to generate more empathetic responses. To this end, we propose a novel framework that improves empathetic response generation by recognizing emotion cause in conversations. Specifically, an emotion reasoner is designed to predict a context emotion label and a sequence of emotion cause-oriented labels, which indicate whether the word is related to the emotion cause. Then we devise both hard and soft gated attention mechanisms to incorporate the emotion cause into response generation. Experiments show that incorporating emotion cause information improves the performance of the model on both emotion recognition and response generation.
We address the problem of language model customization in applications where the ASR component needs to manage domain-specific terminology; although current state-of-the-art speech recognition technology provides excellent results for generic domains
, the adaptation to specialized dictionaries or glossaries is still an open issue. In this work we present an approach for automatically selecting sentences, from a text corpus, that match, both semantically and morphologically, a glossary of terms (words or composite words) furnished by the user. The final goal is to rapidly adapt the language model of an hybrid ASR system with a limited amount of in-domain text data in order to successfully cope with the linguistic domain at hand; the vocabulary of the baseline model is expanded and tailored, reducing the resulting OOV rate. Data selection strategies based on shallow morphological seeds and semantic similarity via word2vec are introduced and discussed; the experimental setting consists in a simultaneous interpreting scenario, where ASRs in three languages are designed to recognize the domainspecific terms (i.e. dentistry). Results using different metrics (OOV rate, WER, precision and recall) show the effectiveness of the proposed techniques.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
