Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userRevisiting the Uniform Information Density Hypothesis
إعادة النظر في فرضية كثافة المعلومات الموحدة
259
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تفرض فرضية كثافة المعلومات الموحدة (UID) تفضيل بين مستخدمي اللغة للكلمات المهنية بحيث يتم توزيع هذه المعلومات بشكل موحد عبر إشارة. في حين أن آثارها على الإنتاج اللغوي قد تم استكشافها جيدا، إلا أن الفرضية يحتمل أن تجعل تنبؤات حول فهم اللغة والقبول اللغوي أيضا. علاوة على ذلك، من غير الواضح كيف ينبغي قياس التوحيد في إشارة لغوية --- أو عدم وجوده --- والوحدة اللغوية، على سبيل المثال، الحكم أو مستوى اللغة، يجب أن تعقد هذه التوحيد. نحن هنا التحقيق في هذه الجوانب من الفرضية UID باستخدام وقت القراءة وبيانات القبول. في حين أن نتائج وقت القراءة لدينا تعمل بشكل عام مع العمل السابق، فإنها تتفق أيضا مع تأثير خطي ضعيف ضعيف، والتي ستكون متوافقة مع توقعات UID. للحصول على أحكام القبول، نجد دليلا أكثر وضوحا على أن عدم التوحيد في كثافة المعلومات هو التنبؤ بانخفاض القبول. ثم نستكشف عمليات تشغيل متعددة من UID، بدافع من تفسيرات مختلفة للفرضية الأصلية، وتحليل النطاق الذي تمارسه الضغط نحو التوحيد. تشير القوة التوضيحية المتمثلة في مجموعة فرعية من عمليات الفرعية المقترحة إلى أن أقوى الاتجاه قد يكون الانحدار نحو مفاجأة يعني في جميع أنحاء اللغة، بدلا من العبارة أو الجملة أو الوثيقة --- اكتشاف يدعم تفسير نموذجي ل UID، أي أنه هو المنتج الثانوي للمستخدمين اللغويين تعظيم استخدام قناة اتصال (افتراضية).
The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal. While its implications on language production have been well explored, the hypothesis potentially makes predictions about language comprehension and linguistic acceptability as well. Further, it is unclear how uniformity in a linguistic signal---or lack thereof---should be measured, and over which linguistic unit, e.g., the sentence or language level, this uniformity should hold. Here we investigate these facets of the UID hypothesis using reading time and acceptability data. While our reading time results are generally consistent with previous work, they are also consistent with a weakly super-linear effect of surprisal, which would be compatible with UID's predictions. For acceptability judgments, we find clearer evidence that non-uniformity in information density is predictive of lower acceptability. We then explore multiple operationalizations of UID, motivated by different interpretations of the original hypothesis, and analyze the scope over which the pressure towards uniformity is exerted. The explanatory power of a subset of the proposed operationalizations suggests that the strongest trend may be a regression towards a mean surprisal across the language, rather than the phrase, sentence, or document---a finding that supports a typical interpretation of UID, namely that it is the byproduct of language users maximizing the use of a (hypothetical) communication channel.
References used
https://aclanthology.org/
rate research
Read More
Pretrained language models have served as the backbone for many state-of-the-art NLP results. These models are large and expensive to train. Recent work suggests that continued pretraining on task-specific data is worth the effort as pretraining lead
s to improved performance on downstream tasks. We explore alternatives to full-scale task-specific pretraining of language models through the use of adapter modules, a parameter-efficient approach to transfer learning. We find that adapter-based pretraining is able to achieve comparable results to task-specific pretraining while using a fraction of the overall trainable parameters. We further explore direct use of adapters without pretraining and find that the direct fine-tuning performs mostly on par with pretrained adapter models, contradicting previously proposed benefits of continual pretraining in full pretraining fine-tuning strategies. Lastly, we perform an ablation study on task-adaptive pretraining to investigate how different hyperparameter settings can change the effectiveness of the pretraining.
When building machine translation systems, one often needs to make the best out of heterogeneous sets of parallel data in training, and to robustly handle inputs from unexpected domains in testing. This multi-domain scenario has attracted a lot of re
cent work that fall under the general umbrella of transfer learning. In this study, we revisit multi-domain machine translation, with the aim to formulate the motivations for developing such systems and the associated expectations with respect to performance. Our experiments with a large sample of multi-domain systems show that most of these expectations are hardly met and suggest that further work is needed to better analyze the current behaviour of multi-domain systems and to make them fully hold their promises.
We describe our two NMT systems submitted to the WMT2021 shared task in English-Czech news translation: CUNI-DocTransformer (document-level CUBBITT) and CUNI-Marian-Baselines. We improve the former with a better sentence-segmentation pre-processing a
nd a post-processing for fixing errors in numbers and units. We use the latter for experiments with various backtranslation techniques.
Recent progress in language modeling has been driven not only by advances in neural architectures, but also through hardware and optimization improvements. In this paper, we revisit the neural probabilistic language model (NPLM) of Bengio et al. (200
3), which simply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. When scaled up to modern hardware, this model (despite its many limitations) performs much better than expected on word-level language model benchmarks. Our analysis reveals that the NPLM achieves lower perplexity than a baseline Transformer with short input contexts but struggles to handle long-term dependencies. Inspired by this result, we modify the Transformer by replacing its first self-attention layer with the NPLM's local concatenation layer, which results in small but consistent perplexity decreases across three word-level language modeling datasets.
Policy gradient algorithms have found wide adoption in NLP, but have recently become subject to criticism, doubting their suitability for NMT. Choshen et al. (2020) identify multiple weaknesses and suspect that their success is determined by the shap
e of output distributions rather than the reward. In this paper, we revisit these claims and study them under a wider range of configurations. Our experiments on in-domain and cross-domain adaptation reveal the importance of exploration and reward scaling, and provide empirical counter-evidence to these claims.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
