| 1 |
Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation ...
|
|
|
|
Abstract:
For multilingual sequence-to-sequence pretrained language models (multilingual Seq2Seq PLMs), e.g. mBART, the self-supervised pretraining task is trained on a wide range of monolingual languages, e.g. 25 languages from commoncrawl, while the downstream cross-lingual tasks generally progress on a bilingual language subset, e.g. English-German, making there exists the cross-lingual data discrepancy, namely \textit{domain discrepancy}, and cross-lingual learning objective discrepancy, namely \textit{task discrepancy}, between the pretrain and finetune stages. To bridge the above cross-lingual domain and task gaps, we extend the vanilla pretrain-finetune pipeline with extra code-switching restore task. Specifically, the first stage employs the self-supervised code-switching restore task as a pretext task, allowing the multilingual Seq2Seq PLM to acquire some in-domain alignment information. And for the second stage, we continuously fine-tune the model on labeled data normally. Experiments on a variety of ...
|
|
Keyword:
Computation and Language cs.CL; FOS Computer and information sciences
|
|
URL: https://dx.doi.org/10.48550/arxiv.2204.07834
https://arxiv.org/abs/2204.07834
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Plurality and Quantification in Graph Representation of Meaning ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 3 |
Plurality and quantification in graph representation of meaning ...
|
|
Cao, Yu. - : No Publisher Supplied, 2021
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Plurality and quantification in graph representation of meaning
|
|
|
|
BASE
|
|
Show details
|
|
|
| 5 |
Automated fact-value distinction in court opinions
|
|
|
|
In: ISSN: 0929-1261 ; EISSN: 1572-9990 ; European Journal of Law and Economics ; https://hal.archives-ouvertes.fr/hal-03174376 ; European Journal of Law and Economics, Springer Verlag, 2020, 50 (3), pp.451-467. ⟨10.1007/s10657-020-09645-7⟩ (2020)
|
|
BASE
|
|
Show details
|
|
|
| 7 |
Automated fact-value distinction in court opinions
|
|
|
|
In: European Journal of Law and Economics, 50 (3) (2020)
|
|
BASE
|
|
Show details
|
|
|
| 9 |
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 10 |
Mastery of Echoics in Chinese Establishes Bidirectional Naming in Chinese for Preschoolers with Naming in English
|
|
|
|
BASE
|
|
Show details
|
|
|
| 11 |
The effects of echoic training on the emergence of naming in a second language by monolingual English-speaking preschool children
|
|
|
|
BASE
|
|
Show details
|
|
|
| 12 |
The effects of echoic training on the emergence of naming in a second language by monolingual English-speaking preschool children ...
|
|
Cao, Yu. - : Columbia University, 2016
|
|
BASE
|
|
Show details
|
|
|
|
|
