| 1 |
Transformer-based NMT : modeling, training and implementation
|
|
Xu, Hongfei. - : Saarländische Universitäts- und Landesbibliothek, 2021
|
|
Abstract:
International trade and industrial collaborations enable countries and regions to concentrate their developments on specific industries while making the most of other countries' specializations, which significantly accelerates global development. However, globalization also increases the demand for cross-region communication. Language barriers between many languages worldwide create a challenge for achieving deep collaboration between groups speaking different languages, increasing the need for translation. Language technology, specifically, Machine Translation (MT) holds the promise to enable communication between languages efficiently in real-time with minimal costs. Even though nowadays computers can perform computation in parallel very fast, which provides machine translation users with translations with very low latency, and although the evolution from Statistical Machine Translation (SMT) to Neural Machine Translation (NMT) with the utilization of advanced deep learning algorithms has significantly boosted translation quality, current machine translation algorithms are still far from accurately translating all input. Thus, how to further improve the performance of state-of-the-art NMT algorithm remains a valuable open research question which has received a wide range of attention. In the research presented in this thesis, we first investigate the long-distance relation modeling ability of the state-of-the-art NMT model, the Transformer. We propose to learn source phrase representations and incorporate them into the Transformer translation model, aiming to enhance its ability to capture long-distance dependencies well. Second, though previous work (Bapna et al., 2018) suggests that deep Transformers have difficulty in converging, we empirically find that the convergence of deep Transformers depends on the interaction between the layer normalization and residual connections employed to stabilize its training. We conduct a theoretical study about how to ensure the convergence of Transformers, especially for deep Transformers, and propose to ensure the convergence of deep Transformers by putting the Lipschitz constraint on its parameter initialization. Finally, we investigate how to dynamically determine proper and efficient batch sizes during the training of the Transformer model. We find that the gradient direction gets stabilized with increasing batch size during gradient accumulation. Thus we propose to dynamically adjust batch sizes during training by monitoring the gradient direction change within gradient accumulation, and to achieve a proper and efficient batch size by stopping the gradient accumulation when the gradient direction starts to fluctuate. For our research in this thesis, we also implement our own NMT toolkit, the Neutron implementation of the Transformer and its variants. In addition to providing fundamental features as the basis of our implementations for the approaches presented in this thesis, we support many advanced features from recent cutting-edge research work. Implementations of all our approaches in this thesis are also included and open-sourced in the toolkit. To compare with previous approaches, we mainly conducted our experiments on the data from the WMT 14 English to German (En-De) and English to French (En-Fr) news translation tasks, except when studying the convergence of deep Transformers, where we alternated the WMT 14 En-Fr task with the WMT 15 Czech to English (Cs-En) news translation task to compare with Bapna et al. (2018). The sizes of these datasets vary from medium (the WMT 14 En-De, ~ 4.5M sentence pairs) to very large (the WMT 14 En-Fr, ~ 36M sentence pairs), thus we suggest our approaches help improve the translation quality between popular language pairs which are widely used and have sufficient data. ; China Scholarship Council
|
|
Keyword:
ddc:430; ddc:440; ddc:490; ddc:491.8; ddc:600; dynamic batch size; neural machine translation; optimization; parameter initialization; phrase representation; transformer translation model
|
|
URL: https://doi.org/10.22028/D291-34998
http://nbn-resolving.org/urn:nbn:de:bsz:291--ds-349988
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Deep interactive text prediction and quality estimation in translation interfaces
|
|
|
|
In: Hokamp, Christopher M. (2018) Deep interactive text prediction and quality estimation in translation interfaces. PhD thesis, Dublin City University. (2018)
|
|
BASE
|
|
Show details
|
|
|
| 3 |
A Hybrid Machine Translation Framework for an Improved Translation Workflow
|
|
Pal, Santanu. - : Saarländische Universitäts- und Landesbibliothek, 2018
|
|
BASE
|
|
Show details
|
|
|
| 9 |
Pluricentric languages : automatic identification and linguistic variation ; Plurizentrische Sprachen : automatische Spracherkennung und linguistische Variation
|
|
|
|
BASE
|
|
Show details
|
|
|
| 10 |
Statistical post-editing and quality estimation for machine translation systems
|
|
|
|
In: Béchara, Hanna (2014) Statistical post-editing and quality estimation for machine translation systems. Master of Science thesis, Dublin City University. (2014)
|
|
BASE
|
|
Show details
|
|
|
| 11 |
Computer assisted (language) learning (CA(L)L) for the inclusive classroom
|
|
Greene, Cara N.. - : Dublin City University. Centre for Next Generation Localisation (CNGL), 2013. : Dublin City University. National Centre for Language Technology (NCLT), 2013. : Dublin City University. School of Computing, 2013
|
|
In: Greene, Cara N. (2013) Computer assisted (language) learning (CA(L)L) for the inclusive classroom. PhD thesis, Dublin City University. (2013)
|
|
BASE
|
|
Show details
|
|
|
| 12 |
Domain adaptation for statistical machine translation of corporate and user-generated content
|
|
|
|
In: Banerjee, Pratyush (2013) Domain adaptation for statistical machine translation of corporate and user-generated content. PhD thesis, Dublin City University. (2013)
|
|
BASE
|
|
Show details
|
|
|
| 13 |
Definition of interfaces
|
|
|
|
In: Almaghout, Hala, Bicici, Ergun, Doherty, Stephen orcid:0000-0003-0887-1049 , Gaspari, Federico, Groves, Declan, Toral, Antonio orcid:0000-0003-2357-2960 , van Genabith, Josef orcid:0000-0003-1322-7944 , Popović, Maja orcid:0000-0001-8234-8745 and Piperidis, Stelios (2013) Definition of interfaces. Project Report. QTLaunchPad. (2013)
|
|
BASE
|
|
Show details
|
|
|
| 14 |
Mapping the industry I: Findings on translation technologies and quality assessment
|
|
|
|
In: Doherty, Stephen orcid:0000-0003-0887-1049 , Gaspari, Federico, Groves, Declan and van Genabith, Josef orcid:0000-0003-1322-7944 (2013) Mapping the industry I: Findings on translation technologies and quality assessment. Technical Report. GALA. (2013)
|
|
BASE
|
|
Show details
|
|
|
| 15 |
Quality metrics for human and machine translation.
|
|
|
|
In: Doherty, Stephen orcid:0000-0003-4864-5986 , Gaspari, Federico, Groves, Declan, Srivastava, Ankit Kumar and van Genabith, Josef orcid:0000-0003-1322-7944 (2013) Quality metrics for human and machine translation. Project Report. UNSPECIFIED. (2013)
|
|
BASE
|
|
Show details
|
|
|
| 16 |
Detecting grammatical errors with treebank-induced, probabilistic parsers
|
|
|
|
In: Wagner, Joachim orcid:0000-0002-8290-3849 (2012) Detecting grammatical errors with treebank-induced, probabilistic parsers. PhD thesis, Dublin City University. (2012)
|
|
BASE
|
|
Show details
|
|
|
| 17 |
Deep Syntax in Statistical Machine Translation
|
|
Graham, Yvette. - : Dublin City University. National Centre for Language Technology (NCLT), 2011. : Dublin City University. School of Computing, 2011
|
|
In: Graham, Yvette (2011) Deep Syntax in Statistical Machine Translation. PhD thesis, Dublin City University. (2011)
|
|
BASE
|
|
Show details
|
|
|
| 18 |
Treebank-Based Deep Grammar Acquisition for French Probabilistic Parsing Resources
|
|
Schluter, Natalie. - : Dublin City University. National Centre for Language Technology (NCLT), 2011. : Dublin City University. School of Computing, 2011
|
|
In: Schluter, Natalie (2011) Treebank-Based Deep Grammar Acquisition for French Probabilistic Parsing Resources. PhD thesis, Dublin City University. (2011)
|
|
BASE
|
|
Show details
|
|
|
| 19 |
The integration of machine translation and translation memory
|
|
He, Yifan. - : Dublin City University. Centre for Next Generation Localisation (CNGL), 2011. : Dublin City University. School of Computing, 2011
|
|
In: He, Yifan (2011) The integration of machine translation and translation memory. PhD thesis, Dublin City University. (2011)
|
|
BASE
|
|
Show details
|
|
|
| 20 |
Treebank-based automatic acquisition of wide coverage, deep linguistic resources for Japanese
|
|
Oya, Masanori. - : Dublin City University. National Centre for Language Technology (NCLT), 2010. : Dublin City University. School of Computing, 2010
|
|
In: Oya, Masanori (2010) Treebank-based automatic acquisition of wide coverage, deep linguistic resources for Japanese. Master of Science thesis, Dublin City University. (2010)
|
|
BASE
|
|
Show details
|
|
|
|
|
