Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userAn Error Analysis Framework for Shallow Surface Realization
إطار تحليل الأخطاء لإعمال السطح الضحل
514
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
مجردة المقاييس المستخدمة بشكل أساسي لتقييم نماذج توليد اللغة الطبيعية (NLG)، مثل Bleu أو Meteor، تفشل في تقديم معلومات حول تأثير العوامل اللغوية الأداء. التركيز على تحقيق السطح (SR)، ومهمة تحويل شجرة تبعية غير مرتبة في جملة رائعة، نقترح إطارا لتحليل الأخطاء الذي يسمح بتحديد ميزات الإدخال تؤثر على نتائج النماذج. يتكون هذا الإطار من عنصرين رئيسيين: (1) تحليلات الارتباط بين مجموعة واسعة من المقاييس النحوية ومقاييس الأداء القياسية و (2) مجموعة من التقنيات لتحديد البنيات النحوية تلقائيا والتي غالبا ما تحدث مع درجات أداء منخفضة. نوضح مزايا إطار الإطار الخاص بنا عن طريق إجراء تحليل الأخطاء في نتائج 174 يدير النظام المقدم إلى المهام المشتركة ل SR متعددة اللغات؛ نظهر أن دقة حافة التبعية ترتبط مع المقاييس التلقائية وبالتالي توفير أساس أكثر قابلية للتفسير للتقييم؛ ونقترح الطرق التي يمكن بها استخدام إطار عملنا لتحسين النماذج والبيانات. يتوفر الإطار في شكل مجموعة أدوات يمكن استخدامها على حد سواء من خلال منظمي الحملة لتوفير ملاحظات مفصلة، من التفسير اللغوي على حالة الفن في مجال الإرسال المتعدد اللغات، والباحثين الفرديين لتحسين النماذج ومجموعات البيانات
Abstract The metrics standardly used to evaluate Natural Language Generation (NLG) models, such as BLEU or METEOR, fail to provide information on which linguistic factors impact performance. Focusing on Surface Realization (SR), the task of converting an unordered dependency tree into a well-formed sentence, we propose a framework for error analysis which permits identifying which features of the input affect the models' results. This framework consists of two main components: (i) correlation analyses between a wide range of syntactic metrics and standard performance metrics and (ii) a set of techniques to automatically identify syntactic constructs that often co-occur with low performance scores. We demonstrate the advantages of our framework by performing error analysis on the results of 174 system runs submitted to the Multilingual SR shared tasks; we show that dependency edge accuracy correlate with automatic metrics thereby providing a more interpretable basis for evaluation; and we suggest ways in which our framework could be used to improve models and data. The framework is available in the form of a toolkit which can be used both by campaign organizers to provide detailed, linguistically interpretable feedback on the state of the art in multilingual SR, and by individual researchers to improve models and datasets.1
References used
https://aclanthology.org/
rate research
Read More
Although deep neural networks have been widely employed and proven effective in sentiment analysis tasks, it remains challenging for model developers to assess their models for erroneous predictions that might exist prior to deployment. Once deployed
, emergent errors can be hard to identify in prediction run-time and impossible to trace back to their sources. To address such gaps, in this paper we propose an error detection framework for sentiment analysis based on explainable features. We perform global-level feature validation with human-in-the-loop assessment, followed by an integration of global and local-level feature contribution analysis. Experimental results show that, given limited human-in-the-loop intervention, our method is able to identify erroneous model predictions on unseen data with high precision.
Traditionally, character-level transduction problems have been solved with finite-state models designed to encode structural and linguistic knowledge of the underlying process, whereas recent approaches rely on the power and flexibility of sequence-t
o-sequence models with attention. Focusing on the less explored unsupervised learning scenario, we compare the two model classes side by side and find that they tend to make different types of errors even when achieving comparable performance. We analyze the distributions of different error classes using two unsupervised tasks as testbeds: converting informally romanized text into the native script of its language (for Russian, Arabic, and Kannada) and translating between a pair of closely related languages (Serbian and Bosnian). Finally, we investigate how combining finite-state and sequence-to-sequence models at decoding time affects the output quantitatively and qualitatively.
Recent research using pre-trained language models for multi-document summarization task lacks deep investigation of potential erroneous cases and their possible application on other languages. In this work, we apply a pre-trained language model (BART
) for multi-document summarization (MDS) task using both fine-tuning and without fine-tuning. We use two English datasets and one German dataset for this study. First, we reproduce the multi-document summaries for English language by following one of the recent studies. Next, we show the applicability of the model to German language by achieving state-of-the-art performance on German MDS. We perform an in-depth error analysis of the followed approach for both languages, which leads us to identifying most notable errors, from made-up facts and topic delimitation, and quantifying the amount of extractiveness.
The variances analysis of direct materials cost in its current image doesn't provide suitable information about the competitive attitude of
economic units from costing side, and doesn't encourage to continuous improvement, and doesn't suitable or su
fficient to modern industrial environment. therefore it should development the traditional analysis tovariances of direct materials cost, paying attention to needs and strategies of modern industrial systems for treatment the criticisms that are directed to this style and improves from role of standard costing system in support of control systems and performance evaluation, and achievement the strategy of continuous improvement.
Where of past, the researcher prepared suggested framework for development of variances analysis of direct materials cost, which suits
with requirements of modern industrial environment, and its application on company of Banias refinery. and the research reaches to series of results, the most important of those are:
The changes which are happened in modern industrial environment didn't led to collapse and disappearance role of standard costing system, and waiving from one of its ways, it is analysis of cost variances.
The traditional analysis to variances of direct materials cost doesn't refer to movements of stock, and doesn't refer to efficiency of the
buying, production and selling processes.
The Reading Machine, is a parsing framework that takes as input raw text and performs six standard nlp tasks: tokenization, pos tagging, morphological analysis, lemmatization, dependency parsing and sentence segmentation. It is built upon Transition
Based Parsing, and allows to implement a large number of parsing configurations, among which a fully incremental one. Three case studies are presented to highlight the versatility of the framework. The first one explores whether an incremental parser is able to take into account top-down dependencies (i.e. the influence of high level decisions on low level ones), the second compares the performances of an incremental and a pipe-line architecture and the third quantifies the impact of the right context on the predictions made by an incremental parser.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
