Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userTowards Objectively Evaluating the Quality of Generated Medical Summaries
نحو تقييم موضوعي جودة الملخصات الطبية الناتجة
388
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نقترح طريقة لتقييم جودة النص الذي تم إنشاؤه عن طريق طلب المقيمين لحساب الحقائق، والحساب الدقيقة، واستدعاء، F-Score، ودقة من التهم الخام.نعتقد أن هذا النهج يؤدي إلى هدف أكثر وأسهل لإعادة إنتاج التقييم.نحن نطبق هذا على مهمة تلخيص التقرير الطبي، حيث قياس الجودة الموضوعية والدقة له أهمية قصوى.
We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.
References used
https://aclanthology.org/
rate research
Read More
Abstract A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics su
ch as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.1
Medical conversations from patient visits are routinely summarized into clinical notes for documentation of clinical care. The automatic creation of clinical note is particularly challenging given that it requires summarization over spoken language a
nd multiple speaker turns; as well, clinical notes include highly technical semi-structured text. In this paper, we describe our corpus creation method and baseline systems for two NLP tasks, clinical dialogue2note sentence alignment and clinical dialogue2note snippet summarization. These two systems, as well as other models created from such a corpus, may be incorporated as parts of an overall end-to-end clinical note generation system.
Text generation systems are ubiquitous in natural language processing applications. However, evaluation of these systems remains a challenge, especially in multilingual settings. In this paper, we propose L'AMBRE -- a metric to evaluate the morphosyn
tactic well-formedness of text using its dependency parse and morphosyntactic rules of the language. We present a way to automatically extract various rules governing morphosyntax directly from dependency treebanks. To tackle the noisy outputs from text generation systems, we propose a simple methodology to train robust parsers. We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.
Extractive summarization has been the mainstay of automatic summarization for decades. Despite all the progress, extractive summarizers still suffer from shortcomings including coreference issues arising from extracting sentences away from their orig
inal context in the source document. This affects the coherence and readability of extractive summaries. In this work, we propose a lightweight post-editing step for extractive summaries that centers around a single linguistic decision: the definiteness of noun phrases. We conduct human evaluation studies that show that human expert judges substantially prefer the output of our proposed system over the original summaries. Moreover, based on an automatic evaluation study, we provide evidence for our system's ability to generate linguistic decisions that lead to improved extractive summaries. We also draw insights about how the automatic system is exploiting some local cues related to the writing style of the main article texts or summary texts to make the decisions, rather than reasoning about the contexts pragmatically.
This paper presents a reference study of available algorithms for plagiarism
detection and it develops semantic plagiarism detection algorithm for plagiarism detection
in medical research papers by employing the Medical Ontologies available on the
World
Wide Web.
The issue of plagiarism detection in medical research written in natural languages is
a complex issue and related exact domain of medical research.
There are many used algorithms for plagiarism detection in natural language, which
are generally divided into two main categories, the first one is comparison algorithms
between files by using fingerprints of files, and files content comparison algorithms, which
include strings matching algorithms and text and tree matching algorithms.
Recently a lot of research in the field of semantic plagiarism detection algorithms
and semantic plagiarism detection algorithms were developed basing of citation analysis
models in scientific research.
In this research a system for plagiarism detection was developed using “Bing” search
engine, where tow type of ontologies used in this system, public ontology as wordNet and
many standard international ontologies in medical domain as Diseases ontology which
contains a descriptions about diseases and definitions of it and the derivation between
diseases.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
