Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userModeling Disclosive Transparency in NLP Application Descriptions
نمذجة الشفافية الكشفية في أوصاف تطبيق NLP
417
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
شفافية إفصائية أوسع --- الحقيقة والوضوح في مجال الاتصالات فيما يتعلق بوظائف منظمة العفو الدولية --- تعتبر على نطاق واسع مرغوب فيه. لسوء الحظ، إنه مفهوم غامض، يصعب تحديده وقياسه. هذا هو الإشكارات، كما أظهر العمل السابق مفاضلات محتملة وعواقب سلبية للإفصاح عن الشفافية، مثل تأثير الالتباس، حيث تفهم الكثير من المعلومات "فهم القارئ لما يعني وصف النظام. لقد جعلت الطبيعة الشخصية الشفافية الإفصائية دراسة عميقة في هذه المشكلات وعلاجاتهم صعبة. لتحسين هذه الحالة، نقدم مقاييس احتمالية النموذج العصبي النموذجي النموذجي مباشرة من الشفافية الكشفية، وإظهار أنهم يرتبطون بالمستخدم وفكرات الخبراء من شفافية النظام، مما يجعلهم وكيلا موضوعيا صالحا. أخيرا، نوضح استخدام هذه المقاييس في دراسة تجريبية يحدد العلاقات بين الشفافية والارتباك وتصورات المستخدمين في كائن من أوصاف نظام NLP الحقيقي.
Broader disclosive transparency---truth and clarity in communication regarding the function of AI systems---is widely considered desirable. Unfortunately, it is a nebulous concept, difficult to both define and quantify. This is problematic, as previous work has demonstrated possible trade-offs and negative consequences to disclosive transparency, such as a confusion effect, where too much information'' clouds a reader's understanding of what a system description means. Disclosive transparency's subjective nature has rendered deep study into these problems and their remedies difficult. To improve this state of affairs, We introduce neural language model-based probabilistic metrics to directly model disclosive transparency, and demonstrate that they correlate with user and expert opinions of system transparency, making them a valid objective proxy. Finally, we demonstrate the use of these metrics in a pilot study quantifying the relationships between transparency, confusion, and user perceptions in a corpus of real NLP system descriptions.
References used
https://aclanthology.org/
rate research
Read More
This paper proposes to study a fine-grained semantic novelty detection task, which can be illustrated with the following example. It is normal that a person walks a dog in the park, but if someone says A man is walking a chicken in the park'', it is
novel. Given a set of natural language descriptions of normal scenes, we want to identify descriptions of novel scenes. We are not aware of any existing work that solves the problem. Although existing novelty or anomaly detection algorithms are applicable, since they are usually topic-based, they perform poorly on our fine-grained semantic novelty detection task. This paper proposes an effective model (called GAT-MA) to solve the problem and also contributes a new dataset. Experimental evaluation shows that GAT-MA outperforms 11 baselines by large margins.
We present a series of programming assignments, adaptable to a range of experience levels from advanced undergraduate to PhD, to teach students design and implementation of modern NLP systems. These assignments build from the ground up and emphasize
full-stack understanding of machine learning models: initially, students implement inference and gradient computation by hand, then use PyTorch to build nearly state-of-the-art neural networks using current best practices. Topics are chosen to cover a wide range of modeling and inference techniques that one might encounter, ranging from linear models suitable for industry applications to state-of-the-art deep learning models used in NLP research. The assignments are customizable, with constrained options to guide less experienced students or open-ended options giving advanced students freedom to explore. All of them can be deployed in a fully autogradable fashion, and have collectively been tested on over 300 students across several semesters.
Text generation from semantic graphs is traditionally performed with deterministic methods, which generate a unique description given an input graph. However, the generation problem admits a range of acceptable textual outputs, exhibiting lexical, sy
ntactic and semantic variation. To address this disconnect, we present two main contributions. First, we propose a stochastic graph-to-text model, incorporating a latent variable in an encoder-decoder model, and its use in an ensemble. Second, to assess the diversity of the generated sentences, we propose a new automatic evaluation metric which jointly evaluates output diversity and quality in a multi-reference setting. We evaluate the models on WebNLG datasets in English and Russian, and show an ensemble of stochastic models produces diverse sets of generated sentences while, retaining similar quality to state-of-the-art models.
Developers of text generation models rely on automated evaluation metrics as a stand-in for slow and expensive manual evaluations. However, image captioning metrics have struggled to give accurate learned estimates of the semantic and pragmatic succe
ss of output text. We address this weakness by introducing the first discourse-aware learned generation metric for evaluating image descriptions. Our approach is inspired by computational theories of discourse for capturing information goals using coherence. We present a dataset of image--description pairs annotated with coherence relations. We then train a coherence-aware metric on a subset of the Conceptual Captions dataset and measure its effectiveness---its ability to predict human ratings of output captions---on a test set composed of out-of-domain images. We demonstrate a higher Kendall Correlation Coefficient for our proposed metric with the human judgments for the results of a number of state-of-the-art coherence-aware caption generation models when compared to several other metrics including recently proposed learned metrics such as BLEURT and BERTScore.
Zero-shot cross-domain dialogue state tracking (DST) enables us to handle unseen domains without the expense of collecting in-domain data. In this paper, we propose a slot descriptions enhanced generative approach for zero-shot cross-domain DST. Spec
ifically, our model first encodes a dialogue context and a slot with a pre-trained self-attentive encoder, and generates slot value in auto-regressive manner. In addition, we incorporate Slot Type Informed Descriptions that capture the shared information of different slots to facilitates the cross-domain knowledge transfer. Experimental results on MultiWOZ shows that our model significantly improve existing state-of-the-art results in zero-shot cross-domain setting.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
