Institutional Repository of Key Laboratory of Behavioral Science, CAS
Investigating inner properties of multimodal representation and semantic compositionality with brain-based componential semantics | |
Wang, Shaonan1,2; Zhang, Jiajun1,2; Lin, Nan3,4![]() | |
2018 | |
Conference Name | 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 |
Source Publication | 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 |
Pages | 5964-5972 |
Conference Date | February 2, 2018 - February 7, 2018 |
Conference Place | New Orleans, LA, United states |
Publisher | AAAI press |
Contribution Rank | 3 |
Abstract | Multimodal models have been proven to outperform text-based approaches on learning semantic representations. However, it still remains unclear what properties are encoded in multimodal representations, in what aspects do they outperform the single-modality representations, and what happened in the process of semantic compositionality in different input modalities. Considering that multimodal models are originally motivated by human concept representations, we assume that correlating multimodal representations with brain-based semantics would interpret their inner properties to answer the above questions. To that end, we propose simple interpretation methods based on brain-based componential semantics. First we investigate the inner properties of multimodal representations by correlating them with corresponding brain-based property vectors. Then we map the distributed vector space to the interpretable brain-based componential space to explore the inner properties of semantic compositionality. Ultimately, the present paper sheds light on the fundamental questions of natural language understanding, such as how to represent the meaning of words and how to combine word meanings into larger units. Copyright © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. |
Keyword | Compositionality - Input modalities - Interpretation methods - Learning semantics - Multimodal models - Natural language understanding - Text-based approach - Word meaning |
Subject Area | Semantics |
ISBN | 9781577358008 |
Indexed By | EI |
Language | 英语 |
EI Accession Number | 20190506435859 |
EI Keywords | Artificial intelligence - Natural language processing systems - Vector spaces |
EI Classification Number | 723.2 Data Processing and Image Processing - 723.4 Artificial Intelligence - 921 Mathematics |
Document Type | 会议论文 |
Identifier | http://ir.psych.ac.cn/handle/311026/30040 |
Collection | 中国科学院行为科学重点实验室 |
Affiliation | 1.National Laboratory of Pattern Recognition, CASIA, Beijing, China; 2.University of Chinese Academy of Sciences, Beijing, China; 3.CAS Key Laboratory of Behavioural Science, Institute of Psychology, Beijing, China; 4.Department of Psychology, University of Chinese Academy of Sciences, Beijing, China; 5.CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China |
Recommended Citation GB/T 7714 | Wang, Shaonan,Zhang, Jiajun,Lin, Nan,et al. Investigating inner properties of multimodal representation and semantic compositionality with brain-based componential semantics[C]:AAAI press,2018:5964-5972. |
Files in This Item: | There are no files associated with this item. |
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment