Skip to Main content Skip to Navigation
Directions of work or proceedings

An Opportunity to Investigate the Role of Specific Nonverbal Cues and First Impression in Interviews using Deepfake Based Controlled Video Generation

Abstract : The study of nonverbal cues in a dyadic interaction, such as a job interview, mostly relies on videos and does not allow to disentangle the role of specific cues. It is thus not clear whether, for instance, an interviewee who smiles while listening to an interviewer would be perceived more favorably than an interviewee who only gazes at an interviewer. While a similar analysis in naturalistic situations requires careful curation of interview recordings, it still does not allow to disentangle the effect of specific nonverbal cues on first impression. Deepfake technology provides the opportunity to address this challenge by creating highly standardized videos of interviewees manifesting a determined behavior (i.e., a combination of specific nonverbal cues). Accordingly, we created a set of deepfake videos enabling us to manipulate the occurrence of three classes of nonverbal attributes (i.e., eye contact, nodding, and smiling). The deepfake videos showed interviewees manifesting one of four behaviors while listening to the interviewer: eye contact with smile and nod, eye contact with only nodding, just eye contact, and looking distracted. Then we tested whether these combinations of nonverbal cues influenced how the interviewees were perceived with respect to personality, confidence, and hireability. Our work reveals the potential of using deepfake technology for generating behaviorally controlled videos, useful for psychology experiments.
Document type :
Directions of work or proceedings
Complete list of metadata

https://hal.univ-reunion.fr/hal-03665990
Contributor : Emmanuelle P. Kleinlogel Connect in order to contact the contributor
Submitted on : Thursday, May 12, 2022 - 11:06:31 AM
Last modification on : Friday, May 13, 2022 - 2:04:50 PM

Identifiers

Collections

Citation

Rahil Satyanarayan Vijay, Kumar Shubham, Laetitia Aurelie Renier, Emmanuelle P. Kleinlogel, Marianne Schmid Mast, et al.. An Opportunity to Investigate the Role of Specific Nonverbal Cues and First Impression in Interviews using Deepfake Based Controlled Video Generation. Hammal Zakia; Busso Carlos; Pelachaud Catherine; Oviatt Sharon; Salah Albert Ali; Zhao Guoying. ICMI '21: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, Oct 2021, Montréal, Canada. ACM, pp.148-152, 2021, ⟨10.1145/3461615.3485397⟩. ⟨hal-03665990⟩

Share

Metrics

Record views

0