Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userA View From the Crowd: Evaluation Challenges for Time-Offset Interaction Applications
عرض من الحشد: تحديات التقييم لتطبيقات تفاعل الزمن
353
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
اكتسبت أنظمة الحوار مثل Chatbots، والمهام مثل الإجابة على الأسئلة (QA) الجر في السنوات الأخيرة؛ بعد تقييم هذه الأنظمة لا تزال صعبة. وتشمل الأسباب مجموعة متنوعة كبيرة في السياقات واستخدام حالات هذه الأنظمة وكذلك التكلفة العالية للتقييم البشري. في هذه الورقة، نركز على نوع معين من أنظمة الحوار: تطبيقات تفاعل إزاحة الوقت (TOIAIA) ذكي، برامج محادثة محاكمة محادثات وجها لوجه بين البشر والحجيجات الإنسانية المسجلة مسبقا. ضمن القيد أن تویا هو نظام إخراج واحد يتفاعل مع المستخدمين مع توقعات مختلفة، نحدد تحديين: أولا، كيف نحدد إجابة جيدة؟ والثاني، ما هو متري مناسب لاستخدامها؟ نستكشف التحديات من خلال إنشاء مجموعة بيانات جديدة تحدد إجابات جيدة متعددة على أسئلة تویا محددة من خلال مساعدة عمال الأمازون الميكانيكية التركيون. يتيح لنا هذه الرأي من الحشد دراسة الاختلافات في كيفية إدراك المحققين الذين يتويا إجاباتها. تشمل مساهماتنا مجموعة البيانات المشروحة التي نجعلها متاحة للجمهور واقتراح معدل النجاح الخاص بك كتقسيط تقييم أكثر ملاءمة من مقاييس استرجاع الجودة التقليدية واسترجاع المعلومات.
Dialogue systems like chatbots, and tasks like question-answering (QA) have gained traction in recent years; yet evaluating such systems remains difficult. Reasons include the great variety in contexts and use cases for these systems as well as the high cost of human evaluation. In this paper, we focus on a specific type of dialogue systems: Time-Offset Interaction Applications (TOIAs) are intelligent, conversational software that simulates face-to-face conversations between humans and pre-recorded human avatars. Under the constraint that a TOIA is a single output system interacting with users with different expectations, we identify two challenges: first, how do we define a good' answer? and second, what's an appropriate metric to use? We explore both challenges through the creation of a novel dataset that identifies multiple good answers to specific TOIA questions through the help of Amazon Mechanical Turk workers. This view from the crowd' allows us to study the variations of how TOIA interrogators perceive its answers. Our contributions include the annotated dataset that we make publicly available and the proposal of Success Rate @k as an evaluation metric that is more appropriate than the traditional QA's and information retrieval's metrics.
References used
https://aclanthology.org/
rate research
Read More
Time-offset interaction applications (TOIA) allow simulating conversations with people who have previously recorded relevant video utterances, which are played in response to their interacting user. TOIAs have great potential for preserving cross-gen
erational and cross-cultural histories, online teaching, simulated interviews, etc. Current TOIAs exist in niche contexts involving high production costs. Democratizing TOIA presents different challenges when creating appropriate pre-recordings, designing different user stories, and creating simple online interfaces for experimentation. We open-source TOIA 2.0, a user-centered time-offset interaction application, and make it available for everyone who wants to interact with people's pre-recordings, or create their pre-recordings.
Services that demanded by users via internet network are classified in two main
kinds, Services work in real time such as video and voice in real time and use UDP
protocol, and other services that work in non-real time such as web browsing (HTTP) a
nd
file transfer (FTP) which use TCP Protocol.
In this research, we study and analyze algorithms that enhance the quality of service
for various applications. For real time application, we use queues disciplines, which gives
high priority for these services and achieves minimum delay. For non-real time application,
we study congestion control algorithms, which achieve best performance for reliable
transfer process with existing the congestion in the network. We used OPNET 14.5
program for simulating various services via internet network. Simulation results show
achieving minimum delay for voice service, and achieving high transmission rate for FTP
application with existing of packets loss in the network.
تطبيقات الزمن الحقيقي
التحكم بالازدحام
Congestion Control
تطبيقات الزمن غير الحقيقي
قواعد الرتل
خوارزمية البداية البطيئة
تجنب الازدحام
خوارزمية إعادة الإرسال السريع و الاستعادة السريعة
Real Time application
non-real time application
Queue disciplines
Slow start algorithms
Congestion avoidance algorithms
Fast retransmit and Fast recovery
المزيد..
The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted evaluation techniques for this ta
sk, using very different approaches and techniques. The best-performing submissions did encouragingly well at this difficult task. However, all automatic submissions struggled to detect factual errors which are semantically or pragmatically complex (for example, based on incorrect computation or inference).
This research introduces a new approach to reduce time execution
of processing programs, by reducing the amount of processed data,
especially in applications where the priority is to the execution time
of the program over the detailed information of captured pictures,
such as detection and tracking systems.
Crowdsourcing has been ubiquitously used for annotating enormous collections of data. However, the major obstacles to using crowd-sourced labels are noise and errors from non-expert annotations. In this work, two approaches dealing with the noise and
errors in crowd-sourced labels are proposed. The first approach uses Sharpness-Aware Minimization (SAM), an optimization technique robust to noisy labels. The other approach leverages a neural network layer called softmax-Crowdlayer specifically designed to learn from crowd-sourced annotations. According to the results, the proposed approaches can improve the performance of the Wide Residual Network model and Multi-layer Perception model applied on crowd-sourced datasets in the image processing domain. It also has similar and comparable results with the majority voting technique when applied to the sequential data domain whereby the Bidirectional Encoder Representations from Transformers (BERT) is used as the base model in both instances.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
