Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userAutomatic generation of a 3D sign language avatar on AR glasses given 2D videos of human signers
الجيل التلقائي من الصورة الرمزية لغة الإشارة ثلاثية الأبعاد في AR نظارات تحتوي على أشرطة الفيديو 2D ل
203
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
في هذه الورقة نقدم التنفيذ النموذجي لخط أنابيب يسمح للجيل التلقائي باللغة الرمزية لغة الإشارة الألمانية من مادة فيديو ثنائية الأبعاد. يرافق العرض التقديمي شفرة المصدر. نقوم بتسجيل حركات تشكل الإنسان أثناء التوقيع مع نماذج رؤية الكمبيوتر. يتم استيراد الإحداثيات المشتركة للأيدي والذراعين كمعالم للسيطرة على الهيكل العظمي الخاص بي. من المعالم المستقلة تشريحية، نقوم بإنشاء هيكل عظمي آخر يعتمد على بنية العظام الهيكلية الرمزية في الصورة الرمزية لحساب بيانات دوران العظام. ثم يتم استخدام هذه البيانات للتحكم في الصورة الرمزية ثلاثية الأبعاد البشرية. يتم عرض الصورة الرمزية على نظارات AR ويمكن وضعها فعليا في الغرفة، بطريقة يمكن اعتبارها في وقت واحد على المتكلم اللفظي. في المزيد من العمل، يهدف إلى تعزيز مع التعرف على الكلام وأساليب ترجمة الآلات للخدمة كترجم بلغة علامة. لقد ثبت أن النموذج الأولي للأشخاص من المجتمع الصم والأشعة السمع لتقييم شؤومها. ظهرت مشاكل مع دورات اليد المنقولة، وكان من الصعب التعرف على إيماءات اليد على الصورة الرمزية بسبب التشوهات مثل شبكات الأصابع الملتوية.
In this paper we present a prototypical implementation of a pipeline that allows the automatic generation of a German Sign Language avatar from 2D video material. The presentation is accompanied by the source code. We record human pose movements during signing with computer vision models. The joint coordinates of hands and arms are imported as landmarks to control the skeleton of our avatar. From the anatomically independent landmarks, we create another skeleton based on the avatar's skeletal bone architecture to calculate the bone rotation data. This data is then used to control our human 3D avatar. The avatar is displayed on AR glasses and can be placed virtually in the room, in a way that it can be perceived simultaneously to the verbal speaker. In further work it is aimed to be enhanced with speech recognition and machine translation methods for serving as a sign language interpreter. The prototype has been shown to people of the deaf and hard-of-hearing community for assessing its comprehensibility. Problems emerged with the transferred hand rotations, hand gestures were hard to recognize on the avatar due to deformations like twisted finger meshes.
References used
https://aclanthology.org/
rate research
Read More
Communication between healthcare professionals and deaf patients is challenging, and the current COVID-19 pandemic makes this issue even more acute. Sign language interpreters can often not enter hospitals and face masks make lipreading impossible. T
o address this urgent problem, we developed a system which allows healthcare professionals to translate sentences that are frequently used in the diagnosis and treatment of COVID-19 into Sign Language of the Netherlands (NGT). Translations are displayed by means of videos and avatar animations. The architecture of the system is such that it could be extended to other applications and other sign languages in a relatively straightforward way.
Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss to-text translation, where a gloss is a sequence of transcribed spoken-language words in the order in which they are signed. We focus here on gloss-to-text
translation, which we treat as a low-resource neural machine translation (NMT) problem. However, unlike traditional low resource NMT, gloss-to-text translation differs because gloss-text pairs often have a higher lexical overlap and lower syntactic overlap than pairs of spoken languages. We exploit this lexical overlap and handle syntactic divergence by proposing two rule-based heuristics that generate pseudo-parallel gloss-text pairs from monolingual spoken language text. By pre-training on this synthetic data, we improve translation from American Sign Language (ASL) to English and German Sign Language (DGS) to German by up to 3.14 and 2.20 BLEU, respectively.
This research aims to explore the potential of low cost video camera for 3D
modelling of large historical monuments. As we know, photos extraction is a fundamental
issue in any photogrammetric project. In fact, cost and time are dependent on photo
extraction method. Usually, photos are taken one by one assuring that every object point is
on two photos at least. This operation is time consuming in large scenes modelling. In the
other hand, video record is a simple operation and requires short time comparing to
traditional photo shooting. Then, it will be useful to suggest an approach to use video
recordings as a source of photos required for 3D modelling.
In the present study, we will evaluate the capability of two video cameras. The first
one is a commercial independent camera and the second is associated with a mobile
phone in the extraction of photos required for 3D modelling of a relatively large objects. It
should be noted that the resolution of video frames in mobile phone cameras (comparing to
professional ones) is less than the resolution of ordinary photos. Hence, 3D models
resulting using these frames will be good for applications that don’t require high precision.
To effectively apply robots in working environments and assist humans, it is essential to develop and evaluate how visual grounding (VG) can affect machine performance on occluded objects. However, current VG works are limited in working environments
, such as offices and warehouses, where objects are usually occluded due to space utilization issues. In our work, we propose a novel OCID-Ref dataset featuring a referring expression segmentation task with referring expressions of occluded objects. OCID-Ref consists of 305,694 referring expressions from 2,300 scenes with providing RGB image and point cloud inputs. To resolve challenging occlusion issues, we argue that it's crucial to take advantage of both 2D and 3D signals to resolve challenging occlusion issues. Our experimental results demonstrate the effectiveness of aggregating 2D and 3D signals but referring to occluded objects still remains challenging for the modern visual grounding systems. OCID-Ref is publicly available at https://github.com/lluma/OCID-Ref
A cascaded Sign Language Translation system first maps sign videos to gloss annotations and then translates glosses into a spoken languages. This work focuses on the second-stage gloss translation component, which is challenging due to the scarcity o
f publicly available parallel data. We approach gloss translation as a low-resource machine translation task and investigate two popular methods for improving translation quality: hyperparameter search and backtranslation. We discuss the potentials and pitfalls of these methods based on experiments on the RWTH-PHOENIX-Weather 2014T dataset.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
