2 |
How transfer learning impacts linguistic knowledge in deep NLP models? ...
|
|
|
|
BASE
|
|
Show details
|
|
3 |
How transfer learning impacts linguistic knowledge in deep NLP models? ...
|
|
|
|
BASE
|
|
Show details
|
|
4 |
Effect of Post-processing on Contextualized Word Representations ...
|
|
|
|
Abstract:
Post-processing of static embedding has beenshown to improve their performance on both lexical and sequence-level tasks. However, post-processing for contextualized embeddings is an under-studied problem. In this work, we question the usefulness of post-processing for contextualized embeddings obtained from different layers of pre-trained language models. More specifically, we standardize individual neuron activations using z-score, min-max normalization, and by removing top principle components using the all-but-the-top method. Additionally, we apply unit length normalization to word representations. On a diverse set of pre-trained models, we show that post-processing unwraps vital information present in the representations for both lexical tasks (such as word similarity and analogy)and sequence classification tasks. Our findings raise interesting points in relation to theresearch studies that use contextualized representations, and suggest z-score normalization as an essential step to consider when using ...
|
|
Keyword:
Artificial Intelligence cs.AI; Computation and Language cs.CL; FOS Computer and information sciences
|
|
URL: https://dx.doi.org/10.48550/arxiv.2104.07456 https://arxiv.org/abs/2104.07456
|
|
BASE
|
|
Hide details
|
|
5 |
Similarity Analysis of Contextual Word Representation Models ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
AraBench: Benchmarking Dialectal Arabic-English Machine Translation ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
A Clustering Framework for Lexical Normalization of Roman Urdu ...
|
|
|
|
BASE
|
|
Show details
|
|
8 |
Analyzing Individual Neurons in Pre-trained Language Models ...
|
|
|
|
BASE
|
|
Show details
|
|
9 |
On the Linguistic Representational Power of Neural Machine Translation Models
|
|
|
|
In: Computational Linguistics, Vol 46, Iss 1, Pp 1-52 (2020) (2020)
|
|
BASE
|
|
Show details
|
|
10 |
On the Linguistic Representational Power of Neural Machine Translation Models ...
|
|
|
|
BASE
|
|
Show details
|
|
11 |
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models ...
|
|
|
|
BASE
|
|
Show details
|
|
12 |
Identifying and Controlling Important Neurons in Neural Machine Translation ...
|
|
|
|
BASE
|
|
Show details
|
|
15 |
Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging ...
|
|
|
|
BASE
|
|
Show details
|
|
16 |
The SUMMA Platform Prototype
|
|
|
|
In: http://infoscience.epfl.ch/record/233575 (2017)
|
|
BASE
|
|
Show details
|
|
17 |
Egyptian Arabic to English Statistical Machine Translation System for NIST OpenMT'2015 ...
|
|
|
|
BASE
|
|
Show details
|
|
18 |
QCMUQ@QALB-2015 Shared Task: Combining Character level MT and Error-tolerant Finite-State Recognition for Arabic Spelling Correction ...
|
|
|
|
BASE
|
|
Show details
|
|
19 |
QCMUQ@QALB-2015 Shared Task: Combining Character level MT and Error-tolerant Finite-State Recognition for Arabic Spelling Correction ...
|
|
|
|
BASE
|
|
Show details
|
|
|
|