| 1 |
How to Evaluate a Summarizer: Study Design and Statistical Analysis for Manual Linguistic Quality Evaluation ...
|
|
|
|
Abstract:
Manual evaluation is essential to judge progress on automatic text summarization. However, we conduct a survey on recent summarization system papers that reveals little agreement on how to perform such evaluation studies. We conduct two evaluation experiments on two aspects of summaries' linguistic quality (coherence and repetitiveness) to compare Likert-type and ranking annotations and show that best choice of evaluation method can vary from one aspect to another. In our survey, we also find that study parameters such as the overall number of annotators and distribution of annotators to annotation items are often not fully reported and that subsequent statistical analysis ignores grouping factors arising from one annotator judging multiple summaries. Using our evaluation experiments, we show that the total number of annotators can have a strong impact on study power and that current statistical analysis methods can inflate type I error rates up to eight-fold. In addition, we highlight that for the purpose ... : Accepted at EACL 2021 ...
|
|
Keyword:
Computation and Language cs.CL; FOS Computer and information sciences
|
|
URL: https://dx.doi.org/10.48550/arxiv.2101.11298
https://arxiv.org/abs/2101.11298
|
|
BASE
|
|
Hide details
|
|
|
| 2 |
Adversarial Datasets for NLI Tasks: the Case of the Chinese Causative-Passive Homonymy ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 4 |
Algorithmus für Kreativität gesucht. Herausforderungen maschinellen Sprachverstehens ...
|
|
|
|
BASE
|
|
Show details
|
|
|
| 7 |
The Impact of News Values and Linguistic Style on the Popularity of Headlines On Twitter and Facebook
|
|
|
|
In: Proceedings of the International AAAI Conference on Web and Social Media; Vol. 11 No. 1 (2017): Eleventh International AAAI Conference on Web and Social Media ; 2334-0770 ; 2162-3449 (2017)
|
|
BASE
|
|
Show details
|
|
|
|
|
