PSYCH OpenIR  > 认知与发展心理学研究室
概念加工在视听跨诵道整合中的作用和制
Alternative TitleThe mechanism of cnncent nrncessing in vidual-language integration
韩海宾
Subtype博士
Thesis Advisor李兴珊
2020-06
Degree Grantor中国科学院心理研究所
Place of Conferral中国科学院心理研究所
Degree Name理学博士
Degree Discipline基础心理学
Keyword语言加工 视觉注意 跨通道整合 概念特征 视觉情境范式
Abstract

人们在日常生活中每时每刻都在接受来自不同通道的信息,不同的认知模块对这些通道的信息进行处理时,往往不是独立工作,而是相互影响。例如,在加工口语和视觉信息时,语言会引导我们的视觉注意,视觉信息也会反馈至语言加工过程,听觉与视觉通道的信息相互影响,相互合作,共同完成整合任务。针对语言和视觉信息跨通道整合现象,研究者们从语义、句法等层面进行了考察。但研究者们对于认知系统如何完成视觉和语言信息的整合任务,仍持有不一致的观点。在本研究中,我们通过两个眼动研究从概念特征角度来系统的考察听觉语言和视觉信息的跨通道整合过程,揭示语言信息和视觉信息的整合机制。我们主要聚焦三个问题:(1)听觉语言信息和视觉信息的整合是否依赖于概念特征的激活,整合过程中的视觉注意转移是否受到概念特征的驱动;(2)概念特征重叠程度是否会影响眼动模式;(3)参与整合的概念特征来源:感知觉输入还是存储的语义知识。

在研究一中,我们首先考察视觉和语言信息整合过程中的视觉注意转移是否依赖于概念特征的提取(特征重叠假说),还是要对物体的名字首先进行提取 (连接假说),并间接考察了概念特征重叠程度对视觉和语言整合过程的影响。Rosch等人(1976)的概念层级为我们提供了一个分离词汇水平与概念特征的方法。其认为概念分为下位概念(麻雀),基本概念(鸟)和上位概念(动物),随着层级的不断提高,所包含的特征与信息不断减少。但是不管是成人还是儿童都发现了基本概念优势效应,即,和物体最为匹配的是基本水平的词汇。在实验1中,以此作为词汇水平和概念特征分离的方法,我们在视觉情境中呈现目标物及三个无关物,并且选取3个对应此目标物的不同等级的概念作为三个不同条件下的听觉目标词。结果发现,概念特征重叠程度越高,注视概率也越高,指向目标物的眼动也更快。实验表明视觉信息与语言信息的整合依赖概念特征的激活,并受到概念特征重叠程度的影响,支持了特征重叠假说。实验2中接续实验1,选取儿童作为被试重复实验1。结果发现了和成人相似的注意模式,进一步支持了概念特征重叠假说。综上,研究一发现视觉语言整合中视觉注意转移依赖于概念特征的提取,整合发生在概念层面,并且间接的表明概念特征重叠程度会影响我们的视觉注意模式。

研究二我们直接操纵了重叠的概念特征,进一步考察了概念特征在视觉与语言整合过程中的作用,以及概念特征的来源。首先,实验3我们选取形状特征作为视觉和听觉语言之间重叠的概念特征,操纵预视时长,考察视觉与语言信息整合过程中的形状效应。结果发现,不管预视时长的长短,眼动都会指向与听到的词汇形状相似的物体。我们采用的目标物图片在视觉上和听觉词汇指明这种整合过程中的形状特征来源于长时记忆中的语义知识。实验4a/4b除了重复实验3的形状效应之外,我们还选取颜色相似的视觉物体和听觉词汇指代物作为颜色条件,以及颜色、形状都相似的视觉物体和听觉词汇指代物作为颜色+形状条件,来考察颜色效应以及多个概念特征的重叠是否要比单个概念特征重叠诱发更多的视觉注视。结果不仅重复了形状效应,而且发现了颜色效应,即,眼动都会指向与听到的词汇颜色相似的物体(例如,“番茄”和“灭火器”)。最重要的是,我们发现颜色+形状条件下目标物的注视概率要大于颜色条件和形状条件,表明视觉和语言表征之间多个概念特征的重叠会比单个特征的重叠诱发更多的眼动。

综上,我们发现:(1>语言和视觉信息的整合依赖于概念特征的激活,该过程发生在概念层面;(2>概念特征重叠程度会对视觉注意模式产生影响,视觉和语言表征之间重叠的概念特征越多,注视概率也越大,为特征重叠假说提供了更直接的证据;(3参与整合的概念特征不仅可以提取自我们存储的语义知识,也同样可接收自感知觉的输入,而且不同的概念特征具有不同的激活程度。通过考察这些问题,我们可以对语言和视觉信息的整合有更深的理解,进一步挖掘了语言和视觉信息整合的内在机制。

Other Abstract

In daily life, people are receiving information from different modalities at all time. A large number of cognitive science studies have shown that different information processing modules do not work independently, but have complex interactions. For example, language guides our visual attention, and visual information can also feedback and affect language processing at the same time. Aiming at revealing the mechanism of the cross-modal integration of linguistic and visual information, most researchers have investigated from different perspectives. But researchers have different opinions. In this study, we systematically examined the cross-modal integration of linguistic and visual information from the perspective of conceptual features by two eye-tracking studies.We focus on three issues: (1) whether the integration of language and visual information depends on the retrieval of conceptual features, and whether the shift of our visual attention is driven by conceptual features during the integration; (2) whether the degree of conceptual overlap affects eye movement patterns; (3) the source of conceptual features involved in integration: perceived information or stored knowledge.

In the first study, we explored whether the integration of visual and linguistic information depends on the extraction of conceptual features and examined how the degree of conceptual overlap affect visual attention indirectly. We adopt Rosch et al.(1976)'s definition of category concept, in which concepts are divided into subordinate concepts (sparrows), basic concepts (birds), and superordinate concepts (animals) according to their abstraction. In Experiment 1,we manipulated the degree of conceptual overlap by varying the spoken words across different hierarchical levels and found that participants looked at the target objects more often when there was greater featural overlap between visual objects and spoken words, suggesting that the degree of conceptual overlap between spoken words and visual objects is the major factor that determines the probability that people look at the target object. Our results supported the Conceptual Overlap Hypothesis. In experiment 2, children were selected as subjects to repeat experiment 1 .The results found similar pattern of visual attention as adults. In summary, the first study found that visual and linguistic information integrates at the conceptual level, and the degree of conceptual overlap affect our visual attention.

In the second study, we manipulated overlapping conceptual features between visual and linguistic information directly, and further examined the source of conceptual features during the integration. First, in Experiment 3, we manipulated the preview duration, and examined the shape effect in the integration of visual and linguistic information. It was found that regardless of the preview duration, our eyes would fixate the objects that have the same shape as the words they heard. Moreover, the shape of target picture we used was different from the shape of referents of spoken words (e.g., a closed "umbrella" and the spoken word "mushroom"), and results indicated that the shape features can be retrieved from semantic knowledge in long-term memory during the integration of visual and linguistic information. In Experiment 4a/4b, in addition to repeating the shape effect of Experiment 3, we also selected visual objects and spoken words with similar colors as the color condition, and with the similar color and shape at the same time as the color+shape condition. What we interested in is the color effect and whether multiple overlapping conceptual features would induce more visual attention than the overlap of one feature. As a result, not only the shape effect was repeated, but also the color effect was found, that is, eye movements would point to objects with similar colors to the spoken words (for example, picture "tomato" and spoken word "fire extinguisher"). Most importantly, we found that the fixation probability under color+shape condition was greater than either color or shape conditions, indicating multiple overlapping conceptual features between visual and linguistic representations will cause more eye movements.

In summary, we found that: (1) language and visual information integrates at the conceptual level and depends on the activation of conceptual features; (2) the degree of conceptual overlap will affect the visual attention during the integration of visual and linguistic information. The more common conceptual features, the greater the fixation probability, providing a direct evidence for the conceptual overlap hypothesis; (3) concept features during the integration can not only be extracted from our stored semantic knowledge, but also can receive from the perceived input. Also, different conceptual features have different activation degrees. By examining these issues, we can have a deeper understanding of the integration of language and visual information, and provide a theoretical support for the further studies from different fields.

Pages94
Language中文
Document Type学位论文
Identifierhttp://ir.psych.ac.cn/handle/311026/31774
Collection认知与发展心理学研究室
Recommended Citation
GB/T 7714
韩海宾. 概念加工在视听跨诵道整合中的作用和制[D]. 中国科学院心理研究所. 中国科学院心理研究所,2020.
Files in This Item:
File Name/Size DocType Version Access License
韩海宾-博士学位论文.pdf(2437KB)学位论文 限制开放CC BY-NC-SAApplication Full Text
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[韩海宾]'s Articles
Baidu academic
Similar articles in Baidu academic
[韩海宾]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[韩海宾]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.