PSYCH OpenIR  > 中国科学院行为科学重点实验室
Investigating inner properties of multimodal representation and semantic compositionality with brain-based componential semantics
Wang, Shaonan1,2; Zhang, Jiajun1,2; Lin, Nan3,4; Zong, Chengqing1,2,5
First AuthorWang, Shaonan
2018
Conference Name32nd AAAI Conference on Artificial Intelligence, AAAI 2018
Source Publication32nd AAAI Conference on Artificial Intelligence, AAAI 2018
Pages5964-5972
Conference DateFebruary 2, 2018 - February 7, 2018
Conference PlaceNew Orleans, LA, United states
PublisherAAAI press
Contribution Rank3
Abstract

Multimodal models have been proven to outperform text-based approaches on learning semantic representations. However, it still remains unclear what properties are encoded in multimodal representations, in what aspects do they outperform the single-modality representations, and what happened in the process of semantic compositionality in different input modalities. Considering that multimodal models are originally motivated by human concept representations, we assume that correlating multimodal representations with brain-based semantics would interpret their inner properties to answer the above questions. To that end, we propose simple interpretation methods based on brain-based componential semantics. First we investigate the inner properties of multimodal representations by correlating them with corresponding brain-based property vectors. Then we map the distributed vector space to the interpretable brain-based componential space to explore the inner properties of semantic compositionality. Ultimately, the present paper sheds light on the fundamental questions of natural language understanding, such as how to represent the meaning of words and how to combine word meanings into larger units. Copyright © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

KeywordCompositionality - Input modalities - Interpretation methods - Learning semantics - Multimodal models - Natural language understanding - Text-based approach - Word meaning
Subject AreaSemantics
ISBN9781577358008
Indexed ByEI
Language英语
EI Accession Number20190506435859
EI KeywordsArtificial intelligence - Natural language processing systems - Vector spaces
EI Classification Number723.2 Data Processing and Image Processing - 723.4 Artificial Intelligence - 921 Mathematics
Document Type会议论文
Identifierhttp://ir.psych.ac.cn/handle/311026/30040
Collection中国科学院行为科学重点实验室
Affiliation1.National Laboratory of Pattern Recognition, CASIA, Beijing, China;
2.University of Chinese Academy of Sciences, Beijing, China;
3.CAS Key Laboratory of Behavioural Science, Institute of Psychology, Beijing, China;
4.Department of Psychology, University of Chinese Academy of Sciences, Beijing, China;
5.CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China
Recommended Citation
GB/T 7714
Wang, Shaonan,Zhang, Jiajun,Lin, Nan,et al. Investigating inner properties of multimodal representation and semantic compositionality with brain-based componential semantics[C]:AAAI press,2018:5964-5972.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Wang, Shaonan]'s Articles
[Zhang, Jiajun]'s Articles
[Lin, Nan]'s Articles
Baidu academic
Similar articles in Baidu academic
[Wang, Shaonan]'s Articles
[Zhang, Jiajun]'s Articles
[Lin, Nan]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Wang, Shaonan]'s Articles
[Zhang, Jiajun]'s Articles
[Lin, Nan]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.