| 1 |
Analysis and Insights from the PARSEME Shared Task dataset ; Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop
|
|
|
|
Abstract:
PUBLISHED ; The PARSEME Shared Task on the automatic identification of verbal multiword expressions (VMWEs) was the first collaborative study on the subject to cover a wide and diverse range of languages. One observation that emerged from the official results is that participating systems performed similarly on each language but differently across languages. That is, intra-language evaluation scores are relatively similar whereas inter-language scores are quite different. We hypothesise that this pattern cannot be attributed solely to the intrinsic linguistic properties in each language corpus, but also to more practical aspects such as the evaluation framework, characteristics of the test and training sets as well as metrics used for measuring performance. This chapter takes a close look at the shared task dataset and the systems? output to explain this pattern. In this process, we produce evaluation results for the systems on VMWEs that only appear in the test set and contrast them with the official evaluation results, which include VMWEs that also occur in the training set. Additionally, we conduct an analysis aimed at estimating the relative difficulty of VMWE detection for each language. This analysis consists of a) assessing the impact on performance of the ability, or lack-thereof, of systems to handle discontinuous and overlapped VMWEs, b) measuring the relative sparsity of sentences with at least one VMWE, and c) interpreting the performance of each system with respect to two baseline systems: a system that simply tags every verb as a VMWE, and a dictionary lookup system. Based on our data analysis, we assess the suitability of the official evaluation methods, specifically the token-based method, and propose to use Cohen?s kappa score as an additional evaluation method.
|
|
Keyword:
Data Analysis; Languages; Natural Language Processing; PARSEME; Verbal multiword expressions
|
|
URL: http://hdl.handle.net/2262/91209
https://doi.org/10.5281/zenodo.1469557
http://people.tcd.ie/maldona
http://langsci-press.org/catalog/view/204/1345/1301-1
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Semantic reranking of CRF label sequences for verbal multiword expression identification ; Multiword expressions at length and in depth: Extended papers from the MWE 2017 workshop
|
|
|
|
BASE
|
|
Show details
|
|
|
| 3 |
An empirical study of segment prioritization for incrementally retrained post-editing-based SMT
|
|
|
|
In: Du, Jinhua orcid:0000-0002-3267-4881 , Ankit, Srivastava, Way, Andy orcid:0000-0001-5736-5930 , Maldonado Guerra, Alfredo orcid:0000-0001-8426-5249 and Lewis, David orcid:0000-0002-3503-4644 (2015) An empirical study of segment prioritization for incrementally retrained post-editing-based SMT. In: The Fifteenth MT Summit Conference, 30 Oct-3 Nov 2015, Miami, FL, USA. (2015)
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Linear transformations of semantic spaces for word-sense discrimination and collocation compositionality grading
|
|
|
|
BASE
|
|
Show details
|
|
|
| 5 |
Multi-word expression-sensitive word alignment
|
|
|
|
In: Okita, Tsuyoshi, Maldonado Guerra, Alfredo orcid:0000-0001-8426-5249 , Graham, Yvette and Way, Andy orcid:0000-0001-5736-5930 (2010) Multi-word expression-sensitive word alignment. In: CLIA 2010 - Fourth International Workshop On Cross Lingual Information Access: Computational Linguistics and the Information Need of Multilingual Societies, 28 Augt 2010, Beijing, China. (2010)
|
|
BASE
|
|
Show details
|
|
|
|
|
