Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs Models
Deepfake is one of the applications that is deemed harmful. Deepfakes are a sort of image or video manipulation in which a person’s image is changed or swapped with that of another person’s face using artificial neural networks. Deepfake manipulations may be done with a variety of techniques and app...
Main Authors: | , , , |
---|---|
Format: | Article |
Language: | English |
Published: |
MDPI AG
2021-07-01
|
Series: | Symmetry |
Subjects: | |
Online Access: | https://www.mdpi.com/2073-8994/13/8/1352 |
id |
doaj-c21bcd278e3f4ceebafd99e02513ab5a |
---|---|
record_format |
Article |
spelling |
doaj-c21bcd278e3f4ceebafd99e02513ab5a2021-08-26T14:23:45ZengMDPI AGSymmetry2073-89942021-07-01131352135210.3390/sym13081352Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs ModelsSemih Yavuzkilic0Abdulkadir Sengur1Zahid Akhtar2Kamran Siddique3Department of Electrical and Electronics Engineering, Fırat University, Elazig 23000, TurkeyDepartment of Electrical and Electronics Engineering, Fırat University, Elazig 23000, TurkeyDepartment of Network and Computer Security, State University of New York Polytechnic Institute, Utica, NY 13502, USADepartment of Information and Communication Technology, Xiamen University Malaysia, Sepang 43900, MalaysiaDeepfake is one of the applications that is deemed harmful. Deepfakes are a sort of image or video manipulation in which a person’s image is changed or swapped with that of another person’s face using artificial neural networks. Deepfake manipulations may be done with a variety of techniques and applications. A quintessential countermeasure against deepfake or face manipulation is deepfake detection method. Most of the existing detection methods perform well under symmetric data distributions, but are still not robust to asymmetric datasets variations and novel deepfake/manipulation types. In this paper, for the identification of fake faces in videos, a new multistream deep learning algorithm is developed, where three streams are merged at the feature level using the fusion layer. After the fusion layer, the fully connected, Softmax, and classification layers are used to classify the data. The pre-trained VGG16 model is adopted for transferred CNN1stream. In transfer learning, the weights of the pre-trained CNN model are further used for training the new classification problem. In the second stream (transferred CNN2), the pre-trained VGG19 model is used. Whereas, in the third stream, the pre-trained ResNet18 model is considered. In this paper, a new large-scale dataset (i.e., World Politicians Deepfake Dataset (WPDD)) is introduced to improve deepfake detection systems. The dataset was created by downloading videos of 20 different politicians from YouTube. Over 320,000 frames were retrieved after dividing the downloaded movie into little sections and extracting the frames. Finally, various manipulations were performed to these frames, resulting in seven separate manipulation classes for men and women. In the experiments, three fake face detection scenarios are investigated. First, fake and real face discrimination is studied. Second, seven face manipulations are performed, including age, beard, face swap, glasses, hair color, hairstyle, smiling, and genuine face discrimination. Third, performance of deepfake detection system under novel type of face manipulation is analyzed. The proposed strategy outperforms the prior existing methods. The calculated performance metrics are over 99%.https://www.mdpi.com/2073-8994/13/8/1352deepfakefake face detectionface manipulationsmulti-stream CNNs |
collection |
DOAJ |
language |
English |
format |
Article |
sources |
DOAJ |
author |
Semih Yavuzkilic Abdulkadir Sengur Zahid Akhtar Kamran Siddique |
spellingShingle |
Semih Yavuzkilic Abdulkadir Sengur Zahid Akhtar Kamran Siddique Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs Models Symmetry deepfake fake face detection face manipulations multi-stream CNNs |
author_facet |
Semih Yavuzkilic Abdulkadir Sengur Zahid Akhtar Kamran Siddique |
author_sort |
Semih Yavuzkilic |
title |
Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs Models |
title_short |
Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs Models |
title_full |
Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs Models |
title_fullStr |
Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs Models |
title_full_unstemmed |
Spotting Deepfakes and Face Manipulations by Fusing Features from Multi-Stream CNNs Models |
title_sort |
spotting deepfakes and face manipulations by fusing features from multi-stream cnns models |
publisher |
MDPI AG |
series |
Symmetry |
issn |
2073-8994 |
publishDate |
2021-07-01 |
description |
Deepfake is one of the applications that is deemed harmful. Deepfakes are a sort of image or video manipulation in which a person’s image is changed or swapped with that of another person’s face using artificial neural networks. Deepfake manipulations may be done with a variety of techniques and applications. A quintessential countermeasure against deepfake or face manipulation is deepfake detection method. Most of the existing detection methods perform well under symmetric data distributions, but are still not robust to asymmetric datasets variations and novel deepfake/manipulation types. In this paper, for the identification of fake faces in videos, a new multistream deep learning algorithm is developed, where three streams are merged at the feature level using the fusion layer. After the fusion layer, the fully connected, Softmax, and classification layers are used to classify the data. The pre-trained VGG16 model is adopted for transferred CNN1stream. In transfer learning, the weights of the pre-trained CNN model are further used for training the new classification problem. In the second stream (transferred CNN2), the pre-trained VGG19 model is used. Whereas, in the third stream, the pre-trained ResNet18 model is considered. In this paper, a new large-scale dataset (i.e., World Politicians Deepfake Dataset (WPDD)) is introduced to improve deepfake detection systems. The dataset was created by downloading videos of 20 different politicians from YouTube. Over 320,000 frames were retrieved after dividing the downloaded movie into little sections and extracting the frames. Finally, various manipulations were performed to these frames, resulting in seven separate manipulation classes for men and women. In the experiments, three fake face detection scenarios are investigated. First, fake and real face discrimination is studied. Second, seven face manipulations are performed, including age, beard, face swap, glasses, hair color, hairstyle, smiling, and genuine face discrimination. Third, performance of deepfake detection system under novel type of face manipulation is analyzed. The proposed strategy outperforms the prior existing methods. The calculated performance metrics are over 99%. |
topic |
deepfake fake face detection face manipulations multi-stream CNNs |
url |
https://www.mdpi.com/2073-8994/13/8/1352 |
work_keys_str_mv |
AT semihyavuzkilic spottingdeepfakesandfacemanipulationsbyfusingfeaturesfrommultistreamcnnsmodels AT abdulkadirsengur spottingdeepfakesandfacemanipulationsbyfusingfeaturesfrommultistreamcnnsmodels AT zahidakhtar spottingdeepfakesandfacemanipulationsbyfusingfeaturesfrommultistreamcnnsmodels AT kamransiddique spottingdeepfakesandfacemanipulationsbyfusingfeaturesfrommultistreamcnnsmodels |
_version_ |
1721189749222277120 |