1 |
The surprising vulnerability of automatic content scoring systems to adversarial input ...
|
|
|
|
Abstract:
Automatic content scoring systems are widely used on short answer tasks to save human effort. However, the use of these systems can invite cheating strategies, such as students writing irrelevant answers in the hopes of gaining at least partial credit. We generate adversarial answers for benchmark content scoring datasets based on different methods of increasing sophistication and show that even simple methods lead to a surprising decrease in content scoring performance. As an extreme example, up to 60% of adversarial answers generated from random shuffling of words in real answers are accepted by a state-of-the-art scoring system. In addition to analyzing the vulnerabilities of content scoring systems, we examine countermeasures such as adversarial training and show that these measures improve system robustness against adversarial answers considerably but do not suffice to completely solve the problem. ...
|
|
Keyword:
Computer and Information Science; Natural Language Processing; Neural Network
|
|
URL: https://dx.doi.org/10.48448/zmf5-zh98 https://underline.io/lecture/6260-the-surprising-vulnerability-of-automatic-content-scoring-systems-to-adversarial-input
|
|
BASE
|
|
Hide details
|
|
2 |
Measuring feature diversity in Native Language Identification
|
|
|
|
BASE
|
|
Show details
|
|
5 |
Is getting the right answer just about choosing the right words? The role of syntactically-informed features in short answer scoring ...
|
|
|
|
BASE
|
|
Show details
|
|
6 |
ETS Corpus of Non-Native Written English ; Educational Testing Service Corpus of Non-Native Written English
|
|
|
|
BASE
|
|
Show details
|
|
7 |
LFG without C-structures
|
|
|
|
In: Cetinoglu, Ozlem, Foster, Jennifer orcid:0000-0002-7789-4853 , Nivre, Joakim, Hogan, Deirdre, Cahill, Aoife orcid:0000-0002-3519-7726 and van Genabith, Josef orcid:0000-0003-1322-7944 (2010) LFG without C-structures. In: the 9th International Workshop on Treebanks and Linguistic Theories, 3 - 4 Dec. 2010, Tartu Estonia. (2010)
|
|
BASE
|
|
Show details
|
|
10 |
Wide-coverage deep statistical parsing using automatic dependency structure annotation
|
|
|
|
In: Cahill, Aoife orcid:0000-0002-3519-7726 , Burke, Michael, O'Donovan, Ruth, Riezler, Stefan, van Genabith, Josef orcid:0000-0003-1322-7944 and Way, Andy orcid:0000-0001-5736-5930 (2008) Wide-coverage deep statistical parsing using automatic dependency structure annotation. Computational Linguistics, 34 (1). pp. 81-124. (2008)
|
|
BASE
|
|
Show details
|
|
12 |
Exploiting multi-word units in history-based probabilistic generation
|
|
|
|
In: Hogan, Deirdre, Cafferkey, Conor, Cahill, Aoife orcid:0000-0002-3519-7726 and van Genabith, Josef (2007) Exploiting multi-word units in history-based probabilistic generation. In: EMNLP-CoNLL 2007 - Joint Meeting of the Conference on Empirical Methods in Natural Language Processing and the Conference on Computational Natural Language Learning, 28-30 June 2007, Prague, Czech Republic. (2007)
|
|
BASE
|
|
Show details
|
|
14 |
Robust PCFG-based generation using automatically acquired LFG approximations
|
|
|
|
In: Cahill, Aoife orcid:0000-0002-3519-7726 and van Genabith, Josef (2006) Robust PCFG-based generation using automatically acquired LFG approximations. In: COLING/ACL 2006 - 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, 17-21 July 2006, Sydney, Australia. (2006)
|
|
BASE
|
|
Show details
|
|
15 |
Adapting and developing linguistic resources for question answering
|
|
Judge, John. - : Dublin City University. School of Computing, 2006
|
|
In: Judge, John (2006) Adapting and developing linguistic resources for question answering. PhD thesis, Dublin City University. (2006)
|
|
BASE
|
|
Show details
|
|
16 |
QuestionBank: creating a corpus of parse-annotated questions
|
|
|
|
In: Judge, John, Cahill, Aoife orcid:0000-0002-3519-7726 and van Genabith, Josef (2006) QuestionBank: creating a corpus of parse-annotated questions. In: COLING/ACL 2006 - 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, 17-21 July 2006, Sydney, Australia. (2006)
|
|
BASE
|
|
Show details
|
|
19 |
Large-scale induction and evaluation of lexical resources from the Penn-II and Penn-III treebanks
|
|
|
|
In: O'Donovan, Ruth, Burke, Michael, Cahill, Aoife orcid:0000-0002-3519-7726 , van Genabith, Josef and Way, Andy orcid:0000-0001-5736-5930 (2005) Large-scale induction and evaluation of lexical resources from the Penn-II and Penn-III treebanks. Computational Linguistics, 31 (3). pp. 328-365. ISSN 1530-9312 (2005)
|
|
BASE
|
|
Show details
|
|
20 |
Evaluating automatically acquired f-structures against PropBank
|
|
|
|
In: Burke, Michael, Cahill, Aoife orcid:0000-0002-3519-7726 , van Genabith, Josef and Way, Andy orcid:0000-0001-5736-5930 (2005) Evaluating automatically acquired f-structures against PropBank. In: LFG05 - 10th International Lexical Functional Grammar Conference, 18-20 July 2005, Bergen, Norway. ISBN 1098-6782 (2005)
|
|
BASE
|
|
Show details
|
|
|
|