Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userIs this the end of the gold standard? A straightforward reference-less grammatical error correction metric
هل هذه هي نهاية المعيار الذهبي؟مرجع مباشر - تقليل خطأ نحوي
458
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
من الصعب ترتيب وتقييم أداء أنظمة تصحيح الأخطاء النحوية (GEC)، حيث يمكن إعادة كتابة جملة بطرق صحيحة عديدة. تم استخدام عدد من مقاييس GEC لتقييم أنظمة GEC المقترحة؛ ومع ذلك، يعتمد كل نظام إما مقارنة بنصوص مرجعية واحدة أو أكثر --- في ما يعرف باسم المعيار الذهبي للمقاييس المستندة إلى المرجعة --- أو مجموعة بيانات منفصلة تفوحية لضبط المرجع المرجعية. النظم القائمة المرجعية لها علاقة منخفضة مع الحكم البشري، لا يمكن التقاط جميع الطرق التي يمكن بها تصحيح الجملة، وتتطلب عمل كبيرا لتطوير مجموعة بيانات اختبار. نقترح نظام تقييم GEC المرجعي الذي يرتبط بشدة بالحكم البشري، يحل القضايا المتعلقة باستخدام مرجع، ولا يحتاج إلى مجموعة بيانات مشروح أخرى للضبط. يعتمد النظام المقترح فقط على الأدوات المتاحة بشكل شائع. بالإضافة إلى ذلك، لا تعمل مقاييس مرجعية متاحة حاليا بشكل صحيح عندما يتكرر جزء من الجملة بدلا من المقاييس المستندة إلى المراجع. في نظامنا المقترح، نتطلع إلى معالجة القضايا المتأصلة في المقاييس المرجعية والمقاييس القائمة على المراجع.
It is difficult to rank and evaluate the performance of grammatical error correction (GEC) systems, as a sentence can be rewritten in numerous correct ways. A number of GEC metrics have been used to evaluate proposed GEC systems; however, each system relies on either a comparison with one or more reference texts---in what is known as the gold standard for reference-based metrics---or a separate annotated dataset to fine-tune the reference-less metric. Reference-based systems have a low correlation with human judgement, cannot capture all the ways in which a sentence can be corrected, and require substantial work to develop a test dataset. We propose a reference-less GEC evaluation system that is strongly correlated with human judgement, solves the issues related to the use of a reference, and does not need another annotated dataset for fine-tuning. The proposed system relies solely on commonly available tools. Additionally, currently available reference-less metrics do not work properly when part of a sentence is repeated as opposed to reference-based metrics. In our proposed system, we look to address issues inherent in reference-less metrics and reference-based metrics.
References used
https://aclanthology.org/
rate research
Read More
GECko+ : a Grammatical and Discourse Error Correction Tool We introduce GECko+, a web-based writing assistance tool for English that corrects errors both at the sentence and at the discourse level. It is based on two state-of-the-art models for gramm
ar error correction and sentence ordering. GECko+ is available online as a web application that implements a pipeline combining the two models.
Although grammatical error correction (GEC) has achieved good performance on texts written by learners of English as a second language, performance on low error density domains where texts are written by English speakers of varying levels of proficie
ncy can still be improved. In this paper, we propose a contrastive learning approach to encourage the GEC model to assign a higher probability to a correct sentence while reducing the probability of incorrect sentences that the model tends to generate, so as to improve the accuracy of the model. Experimental results show that our approach significantly improves the performance of GEC models in low error density domains, when evaluated on the benchmark CWEB dataset.
This paper discusses a classification-based approach to machine translation evaluation, as opposed to a common regression-based approach in the WMT Metrics task. Recent machine translation usually works well but sometimes makes critical errors due to
just a few wrong word choices. Our classification-based approach focuses on such errors using several error type labels, for practical machine translation evaluation in an age of neural machine translation. We made additional annotations on the WMT 2015-2017 Metrics datasets with fluency and adequacy labels to distinguish different types of translation errors from syntactic and semantic viewpoints. We present our human evaluation criteria for the corpus development and automatic evaluation experiments using the corpus. The human evaluation corpus will be publicly available upon publication.
This paper investigates how to correct Chinese text errors with types of mistaken, missing and redundant characters, which are common for Chinese native speakers. Most existing models based on detect-correct framework can correct mistaken characters,
but cannot handle missing or redundant characters due to inconsistency between model inputs and outputs. Although Seq2Seq-based or sequence tagging methods provide solutions to the three error types and achieved relatively good results in English context, they do not perform well in Chinese context according to our experiments. In our work, we propose a novel alignment-agnostic detect-correct framework that can handle both text aligned and non-aligned situations and can serve as a cold start model when no annotation data are provided. Experimental results on three datasets demonstrate that our method is effective and achieves a better performance than most recent published models.
Grammatical error correction (GEC) suffers from a lack of sufficient parallel data. Studies on GEC have proposed several methods to generate pseudo data, which comprise pairs of grammatical and artificially produced ungrammatical sentences. Currently
, a mainstream approach to generate pseudo data is back-translation (BT). Most previous studies using BT have employed the same architecture for both the GEC and BT models. However, GEC models have different correction tendencies depending on the architecture of their models. Thus, in this study, we compare the correction tendencies of GEC models trained on pseudo data generated by three BT models with different architectures, namely, Transformer, CNN, and LSTM. The results confirm that the correction tendencies for each error type are different for every BT model. In addition, we investigate the correction tendencies when using a combination of pseudo data generated by different BT models. As a result, we find that the combination of different BT models improves or interpolates the performance of each error type compared with using a single BT model with different seeds.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
