Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userDeveloping a Clinical Language Model for Swedish: Continued Pretraining of Generic BERT with In-Domain Data
تطوير نموذج لغة سريري للسويدية: استمرار الاحتجاج من بيرت عام مع بيانات داخل المجال
380
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
أصبح استخدام نماذج اللغة المحددة مسبقا، التي تم ضبطها بشكل جيد لأداء مهمة محددة من النهر، على نطاق واسع في NLP.ومع ذلك، قد يكون استخدام نموذج لغة عامة في المجالات المتخصصة، دون المستوى شبه الأمثل بسبب الاختلافات في استخدام اللغة والمفردات.في هذه الورقة، يتم التحقيق في ما إذا كان يمكن تحسين نموذج لغة قائم على اللغة السويدية للمجال السريري من خلال استمرار الاحتجاج بالنص السريري.يتم ضبط نماذج اللغة العامة ومجموعة من المجال بشكل جيد وتقييمها على ثلاثة مهام NLP السريرية الممثلة: (1) تحديد المعلومات الصحية المحمية، (2) تعيين رموز تشخيص ICD-10 إلى الملخصات التفريغ، و (3) عدم اليقين على مستوى الجملةتنبؤ.تظهر النتائج أن الاحيلية المستمرة على البيانات داخل المجال تؤدي إلى تحسين الأداء على جميع المهام الثلاثة المصب، مما يشير إلى وجود قيمة مضافة محتملة لنماذج اللغة الخاصة بالمجال ل NLP السريري.
The use of pretrained language models, fine-tuned to perform a specific downstream task, has become widespread in NLP. Using a generic language model in specialized domains may, however, be sub-optimal due to differences in language use and vocabulary. In this paper, it is investigated whether an existing, generic language model for Swedish can be improved for the clinical domain through continued pretraining with clinical text. The generic and domain-specific language models are fine-tuned and evaluated on three representative clinical NLP tasks: (i) identifying protected health information, (ii) assigning ICD-10 diagnosis codes to discharge summaries, and (iii) sentence-level uncertainty prediction. The results show that continued pretraining on in-domain data leads to improved performance on all three downstream tasks, indicating that there is a potential added value of domain-specific language models for clinical NLP.
References used
https://aclanthology.org/
rate research
Read More
We present on-going work of evaluating the, to our knowledge, first large generative language model trained to converse in Swedish, using data from the online discussion forum Flashback. We conduct a human evaluation pilot study that indicates the mo
del is often able to respond to conversations in both a human-like and informative manner, on a diverse set of topics. While data from online forums can be useful to build conversational systems, we reflect on the negative consequences that incautious application might have, and the need for taking active measures to safeguard against them.
Adapter layers are lightweight, learnable units inserted between transformer layers. Recent work explores using such layers for neural machine translation (NMT), to adapt pre-trained models to new domains or language pairs, training only a small set
of parameters for each new setting (language pair or domain). In this work we study the compositionality of language and domain adapters in the context of Machine Translation. We aim to study, 1) parameter-efficient adaptation to multiple domains and languages simultaneously (full-resource scenario) and 2) cross-lingual transfer in domains where parallel data is unavailable for certain language pairs (partial-resource scenario). We find that in the partial resource scenario a naive combination of domain-specific and language-specific adapters often results in catastrophic forgetting' of the missing languages. We study other ways to combine the adapters to alleviate this issue and maximize cross-lingual transfer. With our best adapter combinations, we obtain improvements of 3-4 BLEU on average for source languages that do not have in-domain data. For target languages without in-domain data, we achieve a similar improvement by combining adapters with back-translation. Supplementary material is available at https://tinyurl.com/r66stbxj.
I participated in the WMT shared news translation task and focus on one high resource language pair: English and Chinese (two directions, Chinese to English and English to Chinese). The submitted systems (ZengHuiMT) focus on data cleaning, data selec
tion, back translation and model ensemble. The techniques I used for data filtering and selection include filtering by rules, language model and word alignment. I used a base translation model trained on initial corpus to obtain the target versions of the WMT21 test sets, then I used language models to find out the monolingual data that is most similar to the target version of test set, such monolingual data was then used to do back translation. On the test set, my best submitted systems achieve 35.9 and 32.2 BLEU for English to Chinese and Chinese to English directions respectively, which are quite high for a small model.
With advances in neural language models, the focus of linguistic steganography has shifted from edit-based approaches to generation-based ones. While the latter's payload capacity is impressive, generating genuine-looking texts remains challenging. I
n this paper, we revisit edit-based linguistic steganography, with the idea that a masked language model offers an off-the-shelf solution. The proposed method eliminates painstaking rule construction and has a high payload capacity for an edit-based model. It is also shown to be more secure against automatic detection than a generation-based method while offering better control of the security/payload capacity trade-off.
We study the problem of domain adaptation in Neural Machine Translation (NMT) when domain-specific data cannot be shared due to confidentiality or copyright issues. As a first step, we propose to fragment data into phrase pairs and use a random sampl
e to fine-tune a generic NMT model instead of the full sentences. Despite the loss of long segments for the sake of confidentiality protection, we find that NMT quality can considerably benefit from this adaptation, and that further gains can be obtained with a simple tagging technique.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
