Deception is a common aspect of human social interaction. In psychological research, deception is defined as a deliberate attempt to mislead others. Previous studies have shown that people achieved only a slightly better than chance accuracy, about 54%. Improving deception detection accuracy is now a challenge in research and application. Previous studies had shown that deception detection from multiple cues could achieve 67.86% accuracy. Studies had also revealed that feedback training observers are better at deception detection performance than control observers. However, how do feedback and multiple cues play a role in deception detection remains to be investigated. In the current study, we used the typical lie detection test to investigate the role of feedback and multiple cues in deception detection, and the cognitive mechanisms.
In study 1, we conducted a lie detection test. During the test, participants were randomly assigned to the feedback group or the control group. They were instructed to watch video fragments of 6 models and judge whether the model’s behavior was misleading or not. Results showed that feedback observers had a better performance than control observers. The accuracy of audiovisual cues condition was better than visual cues condition. There was no interaction between feedback and cue modalities.
In study 2, two experiments were conducted to investigate the mechanism of feedback in deception detection. In experiment 2, the participants were also asked to complete a lie detection test. Besides the truth-lie option, they were given an additional choice of indicating their uncertainty. Chi-squared tests were conducted to analyze the frequencies of participants who made uncertain responses more than zero, five and ten in each condition. The results showed that there were significantly less participants who made uncertain responses in the feedback group than in the control group. What’s more, after eliminating the uncertain responses, there was no main effect of feedback in experiment 2. Compared with experiment 1, the results suggested that feedback could reduce the observer’s uncertainty in deception detection. In experiment 3, we used eye-tracking technology to investigate whether feedback could affect participant’s attentional processing of nonverbal visual cues in deception detection. Analysis of fixation counts and fixation duration showed that both feedback observers and control observers allocated much more attentional resources to face than to hand, leg and foot. But the feedback observers had a longer saccade duration than the control observers. These results suggested that the participants relied on the face value mostly in deception detection, and feedback did not affect the observer’s attentional modes. However, feedback might deepen cognitive processing of the nonverbal visual cues.
In study 3, we investigated the mechanism of multimodal cue’s advantage in deception detection from the visual modality and auditory modality separately. In experiment 4, we used eye-tracking technology to investigate the attentional processing of nonverbal visual cues in multimodal cues based deception detection. Results showed that observers had less fixation counts on nonverbal visual cues, and shorter duration on the leg and foot cues in the multimodal cues condition than in the visual cues condition. The results suggested that less attentional resources were allocated to the nonverbal visual cues in the multimodal cues condition. Furthermore, we coded eight paraverbal indicators and constructed a speech database for deception detection research. Then, we did correlation analysis on the observer’s accuracy and the Likert 7 rating of cues used in the lie detection tests, also on the item’s accuracy and the eight paraverbal indicators. Results showed that there was a significant positive relationship between the observer’s accuracy and the paraverbal cues, and a significant negative relationship between the item’s accuracy and repetition.
Overall, the study confirmed the positive roles of feedback and multimodal cues in deception detection and theoretically explored the correlated cognitive mechanisms. The results would contribute to deception detection studies in both theory and practice. Besides, the database could serve the purpose of automatic deception detection studies in computer science.