1 |
Entity-Based Semantic Adequacy for Data-to-Text Generation
|
|
|
|
In: Findings of the Association for Computational Linguistics: EMNLP 2021 ; Proceeddings of the 2021 Conference on Empirical Methods in Natural Language Processing ; https://hal.archives-ouvertes.fr/hal-03461309 ; Proceeddings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021, Punta Cana, Dominican Republic. pp.1530--1540 (2021)
|
|
Abstract:
International audience ; While powerful pre-trained language models have improved the fluency of text generation models, semantic adequacy -the ability to generate text that is semantically faithful to the input- remains an unsolved issue. In this paper, we introduce a novel automatic evaluation metric, Entity-Based Semantic Adequacy, which can be used to assess to what extent generation models that verbalise RDF (Resource Description Framework) graphs produce text that contains mentions of the entities occurring in the RDF input. This is important as RDF subject and object entities make up 2/3 of the input. We use our metric to compare 25 models from the WebNLG Shared Tasks and we examine correlation with results from human evaluations of semantic adequacy. We show that while our metric correlates with human evaluation scores, this correlation varies with the specifics of the human evaluation setup. This suggests that in order to measure the entity-based adequacy of generated texts, an automatic metric such as the one proposed here might be more reliable, as less subjective and more focused on correct verbalisation of the input, than human evaluation measures.
|
|
Keyword:
[INFO.INFO-AI]Computer Science [cs]/Artificial Intelligence [cs.AI]
|
|
URL: https://hal.archives-ouvertes.fr/hal-03461309 https://hal.archives-ouvertes.fr/hal-03461309/document https://hal.archives-ouvertes.fr/hal-03461309/file/2021.findings-emnlp.Faille.pdf
|
|
BASE
|
|
Hide details
|
|
2 |
An Error Analysis Framework for Shallow Surface Realisation
|
|
|
|
In: EISSN: 2307-387X ; Transactions of the Association for Computational Linguistics ; https://hal.archives-ouvertes.fr/hal-03159422 ; Transactions of the Association for Computational Linguistics, The MIT Press, 2021, 9 ; https://www.aclweb.org/anthology/venues/tacl/ (2021)
|
|
BASE
|
|
Show details
|
|
3 |
Discourse-Based Sentence Splitting
|
|
|
|
In: ACL Anthology ; EMNLP 2021 The 2021 Conference on Empirical Methods in Natural Language Processing ; https://hal.archives-ouvertes.fr/hal-03461298 ; EMNLP 2021 The 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021, Punta Cana, Dominican Republic. pp.1530-1540 (2021)
|
|
BASE
|
|
Show details
|
|
4 |
Gathering Information and Engaging the User ComBot : A Task-Based, Serendipitous Dialog Model for Patient-Doctor Interactions
|
|
|
|
In: Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations ; NLPMC 2021 - Second Workshop on Natural Language Processing for Medical Conversations ; https://hal.archives-ouvertes.fr/hal-03461330 ; NLPMC 2021 - Second Workshop on Natural Language Processing for Medical Conversations, Jun 2021, Mexico, Mexico. pp.21-29 ; https://aclanthology.org/2021.nlpmc-1.3 (2021)
|
|
BASE
|
|
Show details
|
|
5 |
Active Learning and Multi-label Classification for Ellipsis and Coreference Detection in Conversational Question-Answering
|
|
|
|
In: 12th International Workshop on Spoken Dialog System Technology (IWSDS 2021) ; https://hal.archives-ouvertes.fr/hal-03533906 ; 12th International Workshop on Spoken Dialog System Technology (IWSDS 2021), Nov 2021, Singapour/Virtual, Singapore (2021)
|
|
BASE
|
|
Show details
|
|
6 |
Entity-Based Semantic Adequacy for Data-to-Text Generation ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
An Error Analysis Framework for Shallow Surface Realisation ...
|
|
|
|
BASE
|
|
Show details
|
|
9 |
Quality of syntactic implication of RL-based sentence summarization
|
|
|
|
In: AAAI Workshop on Engineering Dependable and Secure Machine Learning Systems 2020 ; https://hal.archives-ouvertes.fr/hal-02883327 ; AAAI Workshop on Engineering Dependable and Secure Machine Learning Systems 2020, Feb 2020, New York, United States (2020)
|
|
BASE
|
|
Show details
|
|
10 |
The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task Overview and Evaluation Results (WebNLG+ 2020)
|
|
|
|
In: Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+) ; https://hal.archives-ouvertes.fr/hal-03148418 ; Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), Dec 2020, Dublin/Virtual, Ireland (2020)
|
|
BASE
|
|
Show details
|
|
11 |
Multilingual AMR-to-Text Generation
|
|
|
|
In: 2020 Conference on Empirical Methods in Natural Language Processing ; https://hal.archives-ouvertes.fr/hal-02999676 ; 2020 Conference on Empirical Methods in Natural Language Processing, Nov 2020, Punta Cana, Dominican Republic (2020)
|
|
BASE
|
|
Show details
|
|
12 |
Augmenting Transformers with KNN-Based Composite Memory for Dialog
|
|
|
|
In: EISSN: 2307-387X ; Transactions of the Association for Computational Linguistics ; https://hal.archives-ouvertes.fr/hal-02999678 ; Transactions of the Association for Computational Linguistics, The MIT Press, In press, ⟨10.1162/tacl_a_00356⟩ ; https://transacl.org/index.php/tacl (2020)
|
|
BASE
|
|
Show details
|
|
13 |
Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs
|
|
|
|
In: EISSN: 2307-387X ; Transactions of the Association for Computational Linguistics ; https://hal.archives-ouvertes.fr/hal-03020314 ; Transactions of the Association for Computational Linguistics, The MIT Press, 2020, 8, ⟨10.1162/tacl_a_00332⟩ (2020)
|
|
BASE
|
|
Show details
|
|
14 |
Learning Health-Bots from Training Data that was Automatically Created using Paraphrase Detection and Expert Knowledge
|
|
|
|
In: Proceedings of the 28th Conference on Computational Linguistics ; https://hal.archives-ouvertes.fr/hal-03020294 ; Proceedings of the 28th Conference on Computational Linguistics, Dec 2020, Barcelona, Spain (2020)
|
|
BASE
|
|
Show details
|
|
16 |
Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs ...
|
|
|
|
BASE
|
|
Show details
|
|
17 |
RL extraction of syntax-based chunks for sentence compression
|
|
|
|
In: ICANN 2019 ; https://hal.archives-ouvertes.fr/hal-02323821 ; ICANN 2019, Sep 2019, Munich, Germany. pp.337-347 (2019)
|
|
BASE
|
|
Show details
|
|
18 |
Surface Realisation Using Full Delexicalisation
|
|
|
|
In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) ; 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing ; https://hal.archives-ouvertes.fr/hal-02277069 ; 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Nov 2019, Hong Kong, China. pp.3084-3094, ⟨10.18653/v1/D19-1305⟩ (2019)
|
|
BASE
|
|
Show details
|
|
19 |
Generating Text from Anonymised Structures
|
|
|
|
In: Proceedings of the 12th International Conference on Natural Language Generation ; https://hal.archives-ouvertes.fr/hal-02460312 ; Proceedings of the 12th International Conference on Natural Language Generation, Oct 2019, Hong Kong, China. pp.112 - 117 (2019)
|
|
BASE
|
|
Show details
|
|
20 |
Revisiting the Binary Linearization Technique for Surface Realization
|
|
|
|
In: Proceedings of The 12th International Conference on Natural Language Generation ; https://hal.archives-ouvertes.fr/hal-02460309 ; Proceedings of The 12th International Conference on Natural Language Generation, 2019, pp.268 - 278 (2019)
|
|
BASE
|
|
Show details
|
|
|
|