Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things

doi:10.1109/ACCESS.2020.3029288

PSYCH OpenIR > 认知与发展心理学研究室

	Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things
	Li, Minjia 1; Xie, Lun 1; Lv, Zeping 2; Li, Juan3 ; Wang, Zhiliang 1
通讯作者	Xie, Lun(xielun@ustb.edu.cn)
摘要	The Internet of Things (IoT) technologies such as interconnection and edge computing help emotion recognition to be applied in healthcare, smart education, etc. However, the acquisition and transmission processes may have some situations, such as lost signals and serious interference noise caused by motion, which affect the quality of the received data and limit the performance of IoT emotion detection. We collectively refer to these as invalid data. A multi-step deep (MSD) system is proposed to reliably detect multimodal emotion by the collected records containing invalid data. Semantic compatibility and continuity are utilized to filter out the invalid data. The feature from invalid modal data is replaced through the imputation method to compensate for the impact of invalid data on emotion detection. In this way, the proposed system can automatically process invalid data and improve the recognition performance. Furthermore, considering the spatiotemporal information, the features of video and physiological signals are extracted by specific deep neural networks in the MSD system. The simulation experiments are conducted on a public multimodal database, and the performance of the MSD system measured by the unweighted average recall is better than that of the traditional system. The promising results observed in the experiments verify the potential influence of the proposed system in practical IoT applications.
关键词	Feature extraction Internet of Things Semantics Databases Task analysis Emotion recognition Biomedical monitoring Internet of Things multimodal emotion detection invalid data multi-step deep (MSD) system deep neural networks
	2020
语种	英语
DOI	10.1109/ACCESS.2020.3029288
发表期刊	IEEE ACCESS
ISSN	2169-3536
卷号	8 页码:187208-187221
收录类别	SCI
资助项目	National Key Research and Development Program of China[2018YFC2001700] ; National Natural Science Foundation of China (Normal Project)[61672093] ; Beijing Municipal Natural Science Foundation[L192005]
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS关键词	HEALTH-CARE IOT ; COMPUTATION
WOS研究方向	Computer Science ; Engineering ; Telecommunications
WOS类目	Computer Science, Information Systems ; Engineering, Electrical & Electronic ; Telecommunications
WOS记录号	WOS:000583561000001
资助机构	National Key Research and Development Program of China ; National Natural Science Foundation of China (Normal Project) ; Beijing Municipal Natural Science Foundation
引用统计	被引频次：5[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.psych.ac.cn/handle/311026/33374
专题	认知与发展心理学研究室
通讯作者	Xie, Lun
作者单位	1.Univ Sci & Technol Beijing, Sch Comp & Commun Engn, Beijing 100083, Peoples R China 2.Rehabil Hosp, Natl Rehabil Auxiliary Ctr, Beijing 100176, Peoples R China 3.Chinese Acad Sci, Inst Psychol, Beijing 100101, Peoples R China
推荐引用方式 GB/T 7714	Li, Minjia,Xie, Lun,Lv, Zeping,et al. Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things[J]. IEEE ACCESS,2020,8:187208-187221.
APA	Li, Minjia,Xie, Lun,Lv, Zeping,Li, Juan,&Wang, Zhiliang.(2020).Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things.IEEE ACCESS,8,187208-187221.
MLA	Li, Minjia,et al."Multistep Deep System for Multimodal Emotion Detection With Invalid Data in the Internet of Things".IEEE ACCESS 8(2020):187208-187221.