Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. In this p
aper we tackle a component of this question by examining robustness of models' ability to deploy relevant context information in the face of distracting content. We present models with cloze tasks requiring use of critical context information, and introduce distracting content to test how robustly the models retain and use that critical information for prediction. We also systematically manipulate the nature of these distractors, to shed light on dynamics of models' use of contextual cues. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of distracting but irrelevant content has clear impact in confusing model predictions. In particular, models appear particularly susceptible to factors of semantic similarity and word position. The findings are consistent with the conclusion that LM predictions are driven in large part by superficial contextual cues, rather than by robust representations of context meaning.
One challenge in evaluating visual question answering (VQA) models in the cross-dataset adaptation setting is that the distribution shifts are multi-modal, making it difficult to identify if it is the shifts in visual or language features that play a
key role. In this paper, we propose a semi-automatic framework for generating disentangled shifts by introducing a controllable visual question-answer generation (VQAG) module that is capable of generating highly-relevant and diverse question-answer pairs with the desired dataset style. We use it to create CrossVQA, a collection of test splits for assessing VQA generalization based on the VQA2, VizWiz, and Open Images datasets. We provide an analysis of our generated datasets and demonstrate its utility by using them to evaluate several state-of-the-art VQA systems. One important finding is that the visual shifts in cross-dataset VQA matter more than the language shifts. More broadly, we present a scalable framework for systematically evaluating the machine with little human intervention.
Communicating with humans is challenging for AIs because it requires a shared understanding of the world, complex semantics (e.g., metaphors or analogies), and at times multi-modal gestures (e.g., pointing with a finger, or an arrow in a diagram). We
investigate these challenges in the context of Iconary, a collaborative game of drawing and guessing based on Pictionary, that poses a novel challenge for the research community. In Iconary, a Guesser tries to identify a phrase that a Drawer is drawing by composing icons, and the Drawer iteratively revises the drawing to help the Guesser in response. This back-and-forth often uses canonical scenes, visual metaphor, or icon compositions to express challenging words, making it an ideal test for mixing language and visual/symbolic communication in AI. We propose models to play Iconary and train them on over 55,000 games between human players. Our models are skillful players and are able to employ world knowledge in language models to play with words unseen during training.
We propose an approach to automatically test for originality in generation tasks where no standard automatic measures exist. Our proposal addresses original uses of language, not necessarily original ideas. We provide an algorithm for our approach an
d a run-time analysis. The algorithm, which finds all of the original fragments in a ground-truth corpus and can reveal whether a generated fragment copies an original without attribution, has a run-time complexity of theta(nlogn) where n is the number of sentences in the ground truth.
The ISO/IEC17025 International Standard for Quality and
competence Assurance for ISO/IEC Test and Calibration
Laboratories have been previously known as the ISO Guide 25, but
the current standard is ISO /IEC 17025: 2005.
Through our study, the HadoopOperationTesting software library
has been developed to provide Big Data applications labs with a
mechanism to test their applications in a simulated environment for
the Hadoop environment with a similar mechanism to test
traditional applications using the JUnit library.
IDDQ testing techniques are used to detect the physical defects
such gate oxide shorts,floating gates and bridging faults, and which
happen for the presence of manufacturing faults during the
manufacturing processes of CMOS integrated circuits, wh
ich
cannot be detected by classical logical testing.
This study will put spot light on web applications testing methods
and tools from the security aspects, and we will explain the details
about using these tools, after we have explained the most famous
weak points and vulnerabilities that web appli
cations suffer from.
At the end we will evaluate these tools.
By this study we try to help developers to choose the most
suitable method and tool for their needs.
Unit testing is a practical approach for increasing the correctness
and quality of software; but writing unit test code is exhausting and
tedious job; and requires a great deal of time and effort. So even
with the use of frameworks for writing and
running unit test such as
JUnit this will need a great deal of time and effort. As a
consequence, there is a pressure in writing testing code. So we
present in this paper a new method to generate unit testing
automatically in order to speed up the testing process and reduce
the cost. We have implemented this method on the Java
programming language, where we write a new specification called
JFS describes the behavior of the function in terms of input and
output. This specification is written inside the code class and is
independent of the code, and it can be written before starting the
code phase and thus achieve the principle TDD Test-Driven
Development which is based on written test-first in order to
improve the development process. After writing specification we
will generate test classes for the execution of unit testing (we used
JUnit as framework to execute unit testing) based on the new
specification.
This research aims to present the importance of using statistical methods while establishing a quality
management system in the laboratory according to the requirements of the international standard ISO
17025:2005.
In addition the research describ
es how statistical analysis of the tests results works and includes a
practical study to evaluate the technical competence of the laboratory by using the most common
statistical methods (hypothesis testing) to study the results in a scientific way enables researchers to
identify weaknesses in the laboratory performance, and thus provides it with feedback and technical
advice helping to determine measurement problems and to check the Trueness of tests results.
Finally, the research provides recommendations and proposals such as a necessity of applying practical
methods for monitoring the performance of tests , making sure they meet quality requirements in terms of
trueness and precision , and working to remove the causes that affect the quality of performance during
all phases of testing, these proposals would – if they have been applied – support the laboratory to obtain
the certification in accordance with international standard ISO 17025:2005.
