من المعروف أن نماذج اللغة العصبية لديها سعة عالية لتحفيظ عينات التدريب.قد يكون لهذا تصرفات خصوصية خطيرة عند نماذج التدريب على محتوى المستخدم مثل مراسلات البريد الإلكتروني.يأتي الخصوصية التفاضلية (DP)، وهو خيار شعبي لتدريب النماذج مع ضمانات الخصوصية، بتكاليف كبيرة من حيث تدهور المرافق والتأثير المتباين على المجموعات الفرعية للمستخدمين.في هذا العمل، نقدم طريقتين مع الحفاظ على الخصوصية لنماذج اللغة التدريبية التي تمكن التحسين المشترك للأداة المساعدة والخصوصية من خلال (1) استخدام تمييزي (2) إدراج مصطلح خسائر ثلاثية جديدة.نقارن أساليبنا مع موانئ دبي من خلال تقييم واسع النطاق.نظهر مزايا المتداولين لدينا مع مفاضلة خصوصية فائدة مواتية، تدرب أسرع مع القدرة على الاستفادة من أساليب التحسين الحالية، وضمان علاج موحد للمجموعات الفرعية الممثلة تمثيلا ممثلي.
Neural language models are known to have a high capacity for memorization of training samples. This may have serious privacy im- plications when training models on user content such as email correspondence. Differential privacy (DP), a popular choice to train models with privacy guarantees, comes with significant costs in terms of utility degradation and disparate impact on subgroups of users. In this work, we introduce two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy through (1) the use of a discriminator and (2) the inclusion of a novel triplet-loss term. We compare our methods with DP through extensive evaluation. We show the advantages of our regularizers with favorable utility-privacy trade-off, faster training with the ability to tap into existing optimization approaches, and ensuring uniform treatment of under-represented subgroups.
References used
https://aclanthology.org/
Modern deep learning models for natural language processing rely heavily on large amounts of annotated texts. However, obtaining such texts may be difficult when they contain personal or confidential information, for example, in health or legal domai
Latent Dirichlet allocation (LDA), a widely used topic model, is often employed as a fundamental tool for text analysis in various applications. However, the training process of the LDA model typically requires massive text corpus data. On one hand,
NLP models are vulnerable to data poisoning attacks. One type of attack can plant a backdoor in a model by injecting poisoned examples in training, causing the victim model to misclassify test instances which include a specific pattern. Although defe
The robustness and security of natural language processing (NLP) models are significantly important in real-world applications. In the context of text classification tasks, adversarial examples can be designed by substituting words with synonyms unde
We curated WikiPII, an automatically labeled dataset composed of Wikipedia biography pages, annotated for personal information extraction. Although automatic annotation can lead to a high degree of label noise, it is an inexpensive process and can ge