Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userMulti Task Learning based Framework for Multimodal Classification
إطار التعلم متعدد المهام المعتمد لتصنيف متعدد الوسائط
269
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تهدف التصنيف متعدد الوسائط واسع النطاق إلى التمييز بين مختلف البيانات متعددة الوسائط، وقد لفت الانتباه بشكل كبير منذ العقد الماضي. في هذه الورقة، نقترح إطارا متعدد المهام في مجال التعلم لمهمة التصنيف المتعدد الوسائط، والتي تتكون من فرعين: فرع متعدد الأضوانات متعدد الوسائط وفرع النمذجة متعددة الوسائط المستنتيت بالاهتمام. يمكن أن يتلقى AutoNcoder متعدد الوسائط ميزات متعددة الوسائط والحصول على المعلومات التفاعلية التي تسمى ميزة التشفير متعددة الوسائط، واستخدام هذه الميزة لإعادة تكوين جميع بيانات الإدخال. بالإضافة إلى ذلك، يمكن استخدام ميزة التشفير المتعددة مشروط لإثراء بيانات DataSet RAW، وتحسين أداء مهام المصب (مثل مهمة التصنيف). أما بالنسبة لفرع النمذجة المتعددة الأبعاد القائم على الانتباه، فإننا نوصي أولا آلية الاهتمام لجعل النموذج يركز على الميزات المهمة، ثم نستخدم ميزة التشفير متعددة الوسائط لإثراء معلومات الإدخال، وتحقيق أداء أفضل. نقوم بإجراء تجارب مكثفة على مجموعة بيانات مختلفة، توضح النتائج فعالية الإطار المقترح.
Large-scale multi-modal classification aim to distinguish between different multi-modal data, and it has drawn dramatically attentions since last decade. In this paper, we propose a multi-task learning-based framework for the multimodal classification task, which consists of two branches: multi-modal autoencoder branch and attention-based multi-modal modeling branch. Multi-modal autoencoder can receive multi-modal features and obtain the interactive information which called multi-modal encoder feature, and use this feature to reconstitute all the input data. Besides, multi-modal encoder feature can be used to enrich the raw dataset, and improve the performance of downstream tasks (such as classification task). As for attention-based multimodal modeling branch, we first employ attention mechanism to make the model focused on important features, then we use the multi-modal encoder feature to enrich the input information, achieve a better performance. We conduct extensive experiments on different dataset, the results demonstrate the effectiveness of proposed framework.
References used
https://aclanthology.org/
rate research
Read More
This paper describes our contribution to the WASSA 2021 shared task on Empathy Prediction and Emotion Classification. The broad goal of this task was to model an empathy score, a distress score and the overall level of emotion of an essay written in
response to a newspaper article associated with harm to someone. We have used the ELECTRA model abundantly and also advanced deep learning approaches like multi-task learning. Additionally, we also leveraged standard machine learning techniques like ensembling. Our system achieves a Pearson Correlation Coefficient of 0.533 on sub-task I and a macro F1 score of 0.5528 on sub-task II. We ranked 1st in Emotion Classification sub-task and 3rd in Empathy Prediction sub-task.
We propose a neural event coreference model in which event coreference is jointly trained with five tasks: trigger detection, entity coreference, anaphoricity determination, realis detection, and argument extraction. To guide the learning of this com
plex model, we incorporate cross-task consistency constraints into the learning process as soft constraints via designing penalty functions. In addition, we propose the novel idea of viewing entity coreference and event coreference as a single coreference task, which we believe is a step towards a unified model of coreference resolution. The resulting model achieves state-of-the-art results on the KBP 2017 event coreference dataset.
Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of image
s and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvement in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers using the trained alignment model improve results on VQA over strong baselines. We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hot-swapping indices.
For a computer to naturally interact with a human, it needs to be human-like. In this paper, we propose a neural response generation model with multi-task learning of generation and classification, focusing on emotion. Our model based on BART (Lewis
et al., 2020), a pre-trained transformer encoder-decoder model, is trained to generate responses and recognize emotions simultaneously. Furthermore, we weight the losses for the tasks to control the update of parameters. Automatic evaluations and crowdsourced manual evaluations show that the proposed model makes generated responses more emotionally aware.
The Multimodal Transformer showed to be a competitive model for multimodal tasks involving textual, visual and audio signals. However, as more modalities are involved, its late fusion by concatenation starts to have a negative impact on the model's p
erformance. Besides, interpreting model's predictions becomes difficult, as one would have to look at the different attention activation matrices. In order to overcome these shortcomings, we propose to perform late fusion by adding a GMU module, which effectively allows the model to weight modalities at instance level, improving its performance while providing a better interpretabilty mechanism. In the experiments, we compare our proposed model (MulT-GMU) against the original implementation (MulT-Concat) and a SOTA model tested in a movie genre classification dataset. Our approach, MulT-GMU, outperforms both, MulT-Concat and previous SOTA model.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
