Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userGoldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty
79
0
0.0
(
0
)
Added by
Quan Ze Chen
Publication date
2021
fields
Informatics Engineering
and research's language is
English
Ask ChatGPT about the research
No Arabic abstract
Human ratings have become a crucial resource for training and evaluating machine learning systems. However, traditional elicitation methods for absolute and comparative rating suffer from issues with consistency and often do not distinguish between uncertainty due to disagreement between annotators and ambiguity inherent to the item being rated. In this work, we present Goldilocks, a novel crowd rating elicitation technique for collecting calibrated scalar annotations that also distinguishes inherent ambiguity from inter-annotator disagreement. We introduce two main ideas: grounding absolute rating scales with examples and using a two-step bounding process to establish a range for an items placement. We test our designs in three domains: judging toxicity of online comments, estimating satiety of food depicted in images, and estimating age based on portraits. We show that (1) Goldilocks can improve consistency in domains where interpretation of the scale is not universal, and that (2) representing items with ranges lets us simultaneously capture different sources of uncertainty leading to better estimates of pairwise relationship distributions.
rate research
Read More
Entropic uncertainty is a well-known concept to formulate uncertainty relations for continuous variable quantum systems with finitely many degrees of freedom. Typically, the bounds of such relations scale with the number of oscillator modes, preventing a straight-forward generalization to quantum field theories. In this work, we overcome this difficulty by introducing the notion of a functional relative entropy and show that it has a meaningful field theory limit. We present the first entropic uncertainty relation for a scalar quantum field theory and exemplify its behavior by considering few particle excitations and the thermal state. Also, we show that the relation implies the Robertson-Schrodinger uncertainty relation.
Web pages contain a large variety of information, but are largely designed for use by graphical web browsers. Mobile access to web-based information often requires presenting HTML web pages using channels that are limited in their graphical capabilities such as small-screens or audio-only interfaces. Content transcoding and annotations have been explored as methods for intelligently presenting HTML documents. Much of this work has focused on transcoding for small-screen devices such as are found on PDAs and cell phones. Here, we focus on the use of annotations and transcoding for presenting HTML content through a voice user interface instantiated in VoiceXML. This transcoded voice interface is designed with an assumption that it will not be used for extended web browsing by voice, but rather to quickly gain directed access to information on web pages. We have found repeated structures that are common in the presentation of data on web pages that are well suited for voice presentation and navigation. In this paper, we describe these structures and their use in an annotation system we have implemented that produces a VoiceXML interface to information originally embedded in HTML documents. We describe the transcoding process used to translate HTML into VoiceXML, including transcoding features we have designed to lead to highly usable VoiceXML code.
Emotion recognition algorithms rely on data annotated with high quality labels. However, emotion expression and perception are inherently subjective. There is generally not a single annotation that can be unambiguously declared correct. As a result, annotations are colored by the manner in which they were collected. In this paper, we conduct crowdsourcing experiments to investigate this impact on both the annotations themselves and on the performance of these algorithms. We focus on one critical question: the effect of context. We present a new emotion dataset, Multimodal Stressed Emotion (MuSE), and annotate the dataset using two conditions: randomized, in which annotators are presented with clips in random order, and contextualized, in which annotators are presented with clips in order. We find that contextual labeling schemes result in annotations that are more similar to a speakers own self-reported labels and that labels generated from randomized schemes are most easily predictable by automated systems.
We contribute a method to automate parameter configurations for chart layouts by learning from human preferences. Existing charting tools usually determine the layout parameters using predefined heuristics, producing sub-optimal layouts. People can repeatedly adjust multiple parameters (e.g., chart size, gap) to achieve visually appealing layouts. However, this trial-and-error process is unsystematic and time-consuming, without a guarantee of improvement. To address this issue, we develop Layout Quality Quantifier (LQ2), a machine learning model that learns to score chart layouts from pairwise crowdsourcing data. Combined with optimization techniques, LQ2 recommends layout parameters that improve the charts layout quality. We apply LQ2 on bar charts and conduct user studies to evaluate its effectiveness by examining the quality of layouts it produces. Results show that LQ2 can generate more visually appealing layouts than both laypeople and baselines. This work demonstrates the feasibility and usages of quantifying human preferences and aesthetics for chart layouts.
Quantum uncertainty relations are formulated in terms of relative entropy between distributions of measurement outcomes and suitable reference distributions with maximum entropy. This type of entropic uncertainty relation can be applied directly to observables with either discrete or continuous spectra. We find that a sum of relative entropies is bounded from above in a nontrivial way, which we illustrate with some examples.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
