Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userOn the Usability of Transformers-based Models for a French Question-Answering Task
على قابلية استخدام النماذج القائمة على المحولات لمهمة الإجابة على الأسئلة الفرنسية
338
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
بالنسبة للعديد من المهام، تم تحقيق النتائج الحديثة مع الهندسة المعمارية القائمة على المحولات، مما يؤدي إلى تحول نموذجي في الممارسات من استخدام الهيغات الخاصة بمهام المهام إلى ضبط نماذج اللغة المدربة مسبقا مسبقا. يتكون الاتجاه المستمر في نماذج تدريبية مع كمية متزايدة باستمرار من البيانات والمعلمات، والتي تتطلب موارد كبيرة. يؤدي إلى بحث قوي لتحسين كفاءة الموارد بناء على تحسينات الخوارزمية والتحسينات التي تم تقييمها للغة الإنجليزية فقط. يثير هذا أسئلة حول قابلية استخدامها عند تطبيقها على مشاكل التعلم الصغيرة، والتي تتوفر كمية محدودة من بيانات التدريب، خاصة لمهام لغات أقل من الموارد. يعد الافتقار إلى كورسا الحجم بشكل مناسب عائقا لتطبيق النهج القائمة على التعلم التي يحركها البيانات ونقلها مع حالات عدم الاستقرار قوية. في هذه الورقة، نقوم بإنشاء أحدث من الجهود المكرسة لسهولة الاستخدام النماذج القائمة على المحولات واقتراح تقييم هذه التحسينات بشأن أداء الإجابة على الأسئلة للغة الفرنسية التي لديها عدد قليل من الموارد. نحن نبذة عن عدم الاستقرار المتعلق ندرة البيانات عن طريق التحقيق في استراتيجيات التدريب المختلفة مع تكبير البيانات وتحسين فرط الحرارة ونقل عبر اللغات. نقدم أيضا نموذجا مدمجا جديدا ل Fralbert الفرنسية التي تثبت أنها تنافسية في إعدادات الموارد المنخفضة.
For many tasks, state-of-the-art results have been achieved with Transformer-based architectures, resulting in a paradigmatic shift in practices from the use of task-specific architectures to the fine-tuning of pre-trained language models. The ongoing trend consists in training models with an ever-increasing amount of data and parameters, which requires considerable resources. It leads to a strong search to improve resource efficiency based on algorithmic and hardware improvements evaluated only for English. This raises questions about their usability when applied to small-scale learning problems, for which a limited amount of training data is available, especially for under-resourced languages tasks. The lack of appropriately sized corpora is a hindrance to applying data-driven and transfer learning-based approaches with strong instability cases. In this paper, we establish a state-of-the-art of the efforts dedicated to the usability of Transformer-based models and propose to evaluate these improvements on the question-answering performances of French language which have few resources. We address the instability relating to data scarcity by investigating various training strategies with data augmentation, hyperparameters optimization and cross-lingual transfer. We also introduce a new compact model for French FrALBERT which proves to be competitive in low-resource settings.
References used
https://aclanthology.org/
rate research
Read More
The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semanti
cally similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.
This work demonstrates the development process of a machine learning architecture for inference that can scale to a large volume of requests. We used a BERT model that was fine-tuned for emotion analysis, returning a probability distribution of emoti
ons given a paragraph. The model was deployed as a gRPC service on Kubernetes. Apache Spark was used to perform inference in batches by calling the service. We encountered some performance and concurrency challenges and created solutions to achieve faster running time. Starting with 200 successful inference requests per minute, we were able to achieve as high as 18 thousand successful requests per minute with the same batch job resource allocation. As a result, we successfully stored emotion probabilities for 95 million paragraphs within 96 hours.
The introduction of transformer-based language models has been a revolutionary step for natural language processing (NLP) research. These models, such as BERT, GPT and ELECTRA, led to state-of-the-art performance in many NLP tasks. Most of these mode
ls were initially developed for English and other languages followed later. Recently, several Arabic-specific models started emerging. However, there are limited direct comparisons between these models. In this paper, we evaluate the performance of 24 of these models on Arabic sentiment and sarcasm detection. Our results show that the models achieving the best performance are those that are trained on only Arabic data, including dialectal Arabic, and use a larger number of parameters, such as the recently released MARBERT. However, we noticed that AraELECTRA is one of the top performing models while being much more efficient in its computational cost. Finally, the experiments on AraGPT2 variants showed low performance compared to BERT models, which indicates that it might not be suitable for classification tasks.
This study describes the development of a Portuguese Community-Question Answering benchmark in the domain of Diabetes Mellitus using a Recognizing Question Entailment (RQE) approach. Given a premise question, RQE aims to retrieve semantically similar
, already answered, archived questions. We build a new Portuguese benchmark corpus with 785 pairs between premise questions and archived answered questions marked with relevance judgments by medical experts. Based on the benchmark corpus, we leveraged and evaluated several RQE approaches ranging from traditional information retrieval methods to novel large pre-trained language models and ensemble techniques using learn-to-rank approaches. Our experimental results show that a supervised transformer-based method trained with multiple languages and for multiple tasks (MUSE) outperforms the alternatives. Our results also show that ensembles of methods (stacking) as well as a traditional (light) information retrieval method (BM25) can produce competitive results. Finally, among the tested strategies, those that exploit only the question (not the answer), provide the best effectiveness-efficiency trade-off. Code is publicly available.
Current pre-trained language models have lots of knowledge, but a more limited ability to use that knowledge. Bloom's Taxonomy helps educators teach children how to use knowledge by categorizing comprehension skills, so we use it to analyze and impro
ve the comprehension skills of large pre-trained language models. Our experiments focus on zero-shot question answering, using the taxonomy to provide proximal context that helps the model answer questions by being relevant to those questions. We show targeting context in this manner improves performance across 4 popular common sense question answer datasets.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
