2 |
Finding Structural Knowledge in Multimodal-BERT ...
|
|
|
|
Abstract:
In this work, we investigate the knowledge learned in the embeddings of multimodal-BERT models. More specifically, we probe their capabilities of storing the grammatical structure of linguistic data and the structure learned over objects in visual data. To reach that goal, we first make the inherent structure of language and visuals explicit by a dependency parse of the sentences that describe the image and by the dependencies between the object regions in the image, respectively. We call this explicit visual structure the \textit{scene tree}, that is based on the dependency tree of the language description. Extensive probing experiments show that the multimodal-BERT models do not encode these scene trees.Code available at \url{https://github.com/VSJMilewski/multimodal-probes}. ... : Accepted at ACL 2022 ...
|
|
Keyword:
Computation and Language cs.CL; Computer Vision and Pattern Recognition cs.CV; FOS Computer and information sciences
|
|
URL: https://arxiv.org/abs/2203.09306 https://dx.doi.org/10.48550/arxiv.2203.09306
|
|
BASE
|
|
Hide details
|
|
3 |
Zero-Shot Dependency Parsing with Worst-Case Aware Automated Curriculum Learning ...
|
|
|
|
BASE
|
|
Show details
|
|
4 |
Parsing with Pretrained Language Models, Multiple Datasets, and Dataset Embeddings ...
|
|
|
|
BASE
|
|
Show details
|
|
5 |
A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs ...
|
|
|
|
BASE
|
|
Show details
|
|
7 |
What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions? ...
|
|
|
|
BASE
|
|
Show details
|
|
9 |
Køpsala: Transition-Based Graph Parsing via Efficient Training and Effective Encoding ...
|
|
|
|
BASE
|
|
Show details
|
|
10 |
Comparison by Conversion: Reverse-Engineering UCCA from Syntax and Lexical Semantics ...
|
|
|
|
BASE
|
|
Show details
|
|
11 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
12 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
13 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
14 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
15 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
16 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
17 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
18 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
19 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
20 |
Representation and parsing of multiword expressions: Current trends
|
|
|
|
In: Language Science Press; (2019)
|
|
BASE
|
|
Show details
|
|
|
|