Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userSwords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality
السيوف: معيار للاستبدال المعجمي مع تحسين تغطية البيانات والجودة
55
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نطلق سراح معيار جديد لاستبدال المعجمات، ومهمة العثور على بدائل مناسبة لكلمة مستهدفة في سياق. للكتابة، يمكن أن أنظمة بدائل معجمية مساعدة البشر من خلال اقتراح الكلمات التي لا يستطيع البشر التفكير فيها بسهولة. ومع ذلك، تعتمد المعايير الحالية على التذكير البشري باعتبارها المصدر الوحيد للبيانات، وبالتالي تفتقر إلى تغطية البدائل التي ستكون مفيدة للبشر. علاوة على ذلك، غالبا ما توفر المحن المعلقون بدائل منخفضة الجودة، والتي ليست مناسبة بالفعل في السياق المحدد. نقوم بجمع التغطية المرتفعة والبيانات ذات الجودة العالية عن طريق تأطير الاستبدال المعجمي كمشكلة تصنيف، تسترشد بالحزانة أنه من الأسهل بالنسبة للبشر الحكم على مدى ملاءمة بدائل المرشحين من الذاكرة. تحقيقا لهذه الغاية، نستخدم المرادفات الخالية من السياق لإنتاج المرشحين والاعتماد على الحكم الإنساني لتحديد مدى ملاءمة السياق. مقارنة بأكبر مؤشر سابق، فإن معيار سيوفنا يحتوي على 3X أكبر عدد ممكن من البدائل لكل كلمة مستهدفة لنفس المستوى من الجودة، وبدائلها أكثر ملاءمة (استنادا إلى الحكم الإنساني) لنفس العدد من البدائل.
We release a new benchmark for lexical substitution, the task of finding appropriate substitutes for a target word in a context. For writing, lexical substitution systems can assist humans by suggesting words that humans cannot easily think of. However, existing benchmarks depend on human recall as the only source of data, and therefore lack coverage of the substitutes that would be most helpful to humans. Furthermore, annotators often provide substitutes of low quality, which are not actually appropriate in the given context. We collect higher-coverage and higher-quality data by framing lexical substitution as a classification problem, guided by the intuition that it is easier for humans to judge the appropriateness of candidate substitutes than conjure them from memory. To this end, we use a context-free thesaurus to produce candidates and rely on human judgement to determine contextual appropriateness. Compared to the previous largest benchmark, our Swords benchmark has 3x as many substitutes per target word for the same level of quality, and its substitutes are 1.4x more appropriate (based on human judgement) for the same number of substitutes.
References used
https://aclanthology.org/
rate research
Read More
Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metric
s cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights on the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations we identify the proportion of different categories of factual errors and benchmark factuality metrics, showing their correlation with human judgement as well as their specific strengths and weaknesses.
We introduce a high-quality and large-scale Vietnamese-English parallel dataset of 3.02M sentence pairs, which is 2.9M pairs larger than the benchmark Vietnamese-English machine translation corpus IWSLT15. We conduct experiments comparing strong neur
al baselines and well-known automatic translation engines on our dataset and find that in both automatic and human evaluations: the best performance is obtained by fine-tuning the pre-trained sequence-to-sequence denoising auto-encoder mBART. To our best knowledge, this is the first large-scale Vietnamese-English machine translation study. We hope our publicly available dataset and study can serve as a starting point for future research and applications on Vietnamese-English machine translation. We release our dataset at: https://github.com/VinAIResearch/PhoMT
Authorship attribution is the task of assigning an unknown document to an author from a set of candidates. In the past, studies in this field use various evaluation datasets to demonstrate the effectiveness of preprocessing steps, features, and model
s. However, only a small fraction of works use more than one dataset to prove claims. In this paper, we present a collection of highly diverse authorship attribution datasets, which better generalizes evaluation results from authorship attribution research. Furthermore, we implement a wide variety of previously used machine learning models and show that many approaches show vastly different performances when applied to different datasets. We include pre-trained language models, for the first time testing them in this field in a systematic way. Finally, we propose a set of aggregated scores to evaluate different aspects of the dataset collection.
Recent progress in natural language processing has led to Transformer architectures becoming the predominant model used for natural language tasks. However, in many real- world datasets, additional modalities are included which the Transformer does n
ot directly leverage. We present Multimodal- Toolkit, an open-source Python package to incorporate text and tabular (categorical and numerical) data with Transformers for downstream applications. Our toolkit integrates well with Hugging Face's existing API such as tokenization and the model hub which allows easy download of different pre-trained models.
Challenging problems such as open-domain question answering, fact checking, slot filling and entity linking require access to large, external knowledge sources. While some models do well on individual tasks, developing general models is difficult as
each task might require computationally expensive indexing of custom knowledge sources, in addition to dedicated infrastructure. To catalyze research on models that condition on specific information in large textual resources, we present a benchmark for knowledge-intensive language tasks (KILT). All tasks in KILT are grounded in the same snapshot of Wikipedia, reducing engineering turnaround through the re-use of components, as well as accelerating research into task-agnostic memory architectures. We test both task-specific and general baselines, evaluating downstream performance in addition to the ability of the models to provide provenance. We find that a shared dense vector index coupled with a seq2seq model is a strong baseline, outperforming more tailor-made approaches for fact checking, open-domain question answering and dialogue, and yielding competitive results on entity linking and slot filling, by generating disambiguated text. KILT data and code are available at https://github.com/facebookresearch/KILT.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
