Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLeveraging Community and Author Context to Explain the Performance and Bias of Text-Based Deception Detection Models
الاستفادة من المجتمع والسياق المؤلف لشرح أداء وتحيز نماذج الكشف عن الخداع القائمة على النص
504
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يمكن اكتشاف الوظائف الإخبارية الخادعة المشتركة في المجتمعات عبر الإنترنت مع نماذج NLP، وقد ركزت البحوث الحديثة الكثير على تطوير هذه النماذج.في هذا العمل، نستخدم خصائص المجتمعات والمؤلفين عبر الإنترنت --- سياق كيفية نشر المحتوى - - لشرح أداء نموذج كشف الخداع الشبكي العصبي وتحديد السكان الفرعيين الذين يتأثرون بشكل غير متناسب بدقة نموذجيةأو الفشل.نحن ندرس من يقوم بنشر المحتوى، وحيث يتم نشر المحتوى إليه.نجد أنه في حين أن خصائص المؤلف هي أفضل من المتنبئين من المحتوى الخادع من الخصائص المجتمعية، فإن كلا الخصائص مرتبطة بقوة بأداء نموذجي.قد تفشل مقاييس الأداء التقليدية مثل درجة F1 في التقاط أداء نموذجي ضعيف على السكان الفرعيين المعزولين مثل المؤلفين المحددين، وعلى هذا النحو، فإن التقييم الأكثر دقة لنماذج الكشف عن الخداع أمر بالغ الأهمية.
Deceptive news posts shared in online communities can be detected with NLP models, and much recent research has focused on the development of such models. In this work, we use characteristics of online communities and authors --- the context of how and where content is posted --- to explain the performance of a neural network deception detection model and identify sub-populations who are disproportionately affected by model accuracy or failure. We examine who is posting the content, and where the content is posted to. We find that while author characteristics are better predictors of deceptive content than community characteristics, both characteristics are strongly correlated with model performance. Traditional performance metrics such as F1 score may fail to capture poor model performance on isolated sub-populations such as specific authors, and as such, more nuanced evaluation of deception detection models is critical.
References used
https://aclanthology.org/
rate research
Read More
With the increasing use of machine-learning driven algorithmic judgements, it is critical to develop models that are robust to evolving or manipulated inputs. We propose an extensive analysis of model robustness against linguistic variation in the se
tting of deceptive news detection, an important task in the context of misinformation spread online. We consider two prediction tasks and compare three state-of-the-art embeddings to highlight consistent trends in model performance, high confidence misclassifications, and high impact failures. By measuring the effectiveness of adversarial defense strategies and evaluating model susceptibility to adversarial attacks using character- and word-perturbed text, we find that character or mixed ensemble models are the most effective defenses and that character perturbation-based attack tactics are more successful.
Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and i
nference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. We also conduct experiments on our newly proposed benchmark to show that the augmentation effect is not only attributed to memorization. Further ablation studies and a qualitative analysis provide more insights into our approach.
Identifying emotions from text is crucial for a variety of real world tasks. We consider the two largest now-available corpora for emotion classification: GoEmotions, with 58k messages labelled by readers, and Vent, with 33M writer-labelled messages.
We design a benchmark and evaluate several feature spaces and learning algorithms, including two simple yet novel models on top of BERT that outperform previous strong baselines on GoEmotions. Through an experiment with human participants, we also analyze the differences between how writers express emotions and how readers perceive them. Our results suggest that emotions expressed by writers are harder to identify than emotions that readers perceive. We share a public web interface for researchers to explore our models.
This paper describes our approach (IIITH) for SemEval-2021 Task 5: HaHackathon: Detecting and Rating Humor and Offense. Our results focus on two major objectives: (i) Effect of task adaptive pretraining on the performance of transformer based models
(ii) How does lexical and hurtlex features help in quantifying humour and offense. In this paper, we provide a detailed description of our approach along with comparisions mentioned above.
Learning a good latent representation is essential for text style transfer, which generates a new sentence by changing the attributes of a given sentence while preserving its content. Most previous works adopt disentangled latent representation learn
ing to realize style transfer. We propose a novel text style transfer algorithm with entangled latent representation, and introduce a style classifier that can regulate the latent structure and transfer style. Moreover, our algorithm for style transfer applies to both single-attribute and multi-attribute transfer. Extensive experimental results show that our method generally outperforms state-of-the-art approaches.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
