Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userCorpus Creation and Language Identification in Low-Resource Code-Mixed Telugu-English Text
إنشاء Corpus وتحديد اللغة في النص المنخفض
245
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تعد خلط الكود (سم) ظاهرة شائعة في المجتمعات متعددة اللغات. يلعب CM دورا مهما في مجال التكنولوجيا والحقول الطبية حيث تكون المصطلحات في اللغة الأم وغير معروفة. سيساعد تحديد اللغة (غطاء) من بيانات CM حل مهام NLP مثل التدقيق الإملائي، والتعرف على الكيان المسمى، وعلامات جزء من الكلام، والتحليل الدلالي. في العصر الحالي من التعلم الآلي، فإن المشكلة المشتركة للمهام المذكورة أعلاه هي توافر بيانات التعلم لتدريب نماذج. في هذه الورقة، نقدم مجموعة بيانات Telugu-English-English مزقوفة يدويا (مجموعة بيانات Twitter DataSet and Blog). تحتوي مجموعة بيانات Twitter على المزيد من التباين بالحروف بالحروف اللاتينية وأصحاب أخطاء إملائية من مجموعة بيانات المدونة. قارننا عبر نماذج التصنيف المختلفة وإجراء علامات مقاعد البدلاء واسعة النطاق باستخدام نماذج التعلم الكلاسيكية والعميقة للغطاء مقارنة بالنماذج الحالية. نقترح دورتين لتصنيف اللغة (التيلجو والإنجليزي) في بيانات CM: (1) تصنيف مستوى تصنيف مستوى الكلمات (2) تصنيف الكلمات من Word-Word وتقارن هذه النهج تقديم خطين قويين للغطاء على مجموعات البيانات هذه.
Code-Mixing (CM) is a common phenomenon in multilingual societies. CM plays a significant role in technology and medical fields where terminologies in the native language are not available or known. Language Identification (LID) of the CM data will help solve NLP tasks such as Spell Checking, Named Entity Recognition, Part-Of-Speech tagging, and Semantic Parsing. In the current era of machine learning, a common problem to the above-mentioned tasks is the availability of Learning data to train models. In this paper, we introduce two Telugu-English CM manually annotated datasets (Twitter dataset and Blog dataset). The Twitter dataset contains more romanization variability and misspelled words than the blog dataset. We compare across various classification models and perform extensive bench-marking using both Classical and Deep Learning Models for LID compared to existing models. We propose two architectures for language classification (Telugu and English) in CM data: (1) Word Level Classification (2) Sentence Level word-by-word Classification and compare these approaches presenting two strong baselines for LID on these datasets.
References used
https://aclanthology.org/
rate research
Read More
The National Virtual Translation Center (NVTC) seeks to acquire human language technology (HLT) tools that will facilitate its mission to provide verbatim English translations of foreign language audio and video files. In the text domain, NVTC has be
en using translation memory (TM) for some time and has reported on the incorporation of machine translation (MT) into that workflow (Miller et al., 2020). While we have explored the use of speech-totext (STT) and speech translation (ST) in the past (Tzoukermann and Miller, 2018), we have now invested in the creation of a substantial human-made corpus to thoroughly evaluate alternatives. Results from our analysis of this corpus and the performance of HLT tools point the way to the most promising ones to deploy in our workflow.
Text simplification is a growing field with many potential useful applications. Training text simplification algorithms generally requires a lot of annotated data, however there are not many corpora suitable for this task. We propose a new unsupervis
ed method for aligning text based on Doc2Vec embeddings and a new alignment algorithm, capable of aligning texts at different levels. Initial evaluation shows promising results for the new approach. We used the newly developed approach to create a new monolingual parallel corpus composed of the works of English early modern philosophers and their corresponding simplified versions.
Meta-learning has achieved great success in leveraging the historical learned knowledge to facilitate the learning process of the new task. However, merely learning the knowledge from the historical tasks, adopted by current meta-learning algorithms,
may not generalize well to testing tasks when they are not well-supported by training tasks. This paper studies a low-resource text classification problem and bridges the gap between meta-training and meta-testing tasks by leveraging the external knowledge bases. Specifically, we propose KGML to introduce additional representation for each sentence learned from the extracted sentence-specific knowledge graph. The extensive experiments on three datasets demonstrate the effectiveness of KGML under both supervised adaptation and unsupervised adaptation settings.
Taxonomies are symbolic representations of hierarchical relationships between terms or entities. While taxonomies are useful in broad applications, manually updating or maintaining them is labor-intensive and difficult to scale in practice. Conventio
nal supervised methods for this enrichment task fail to find optimal parents of new terms in low-resource settings where only small taxonomies are available because of overfitting to hierarchical relationships in the taxonomies. To tackle the problem of low-resource taxonomy enrichment, we propose Musubu, an efficient framework for taxonomy enrichment in low-resource settings with pretrained language models (LMs) as knowledge bases to compensate for the shortage of information. Musubu leverages an LM-based classifier to determine whether or not inputted term pairs have hierarchical relationships. Musubu also utilizes Hearst patterns to generate queries to leverage implicit knowledge from the LM efficiently for more accurate prediction. We empirically demonstrate the effectiveness of our method in extensive experiments on taxonomies from both a SemEval task and real-world retailer datasets.
In this shared task, we seek the participating teams to investigate the factors influencing the quality of the code-mixed text generation systems. We synthetically generate code-mixed Hinglish sentences using two distinct approaches and employ human
annotators to rate the generation quality. We propose two subtasks, quality rating prediction and annotators' disagreement prediction of the synthetic Hinglish dataset. The proposed subtasks will put forward the reasoning and explanation of the factors influencing the quality and human perception of the code-mixed text.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
