notesum.ai

Published at December 5

SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Model

cs.CV

cs.AI

Released Date: December 5, 2024

Authors: Zhenglin Huang¹, Jinwei Hu¹, Xiangtai Li², Yiwei He¹, Xingyu Zhao³, Bei Peng¹, Baoyuan Wu⁴, Xiaowei Huang¹, Guangliang Cheng¹

Aff.: ¹University of Liverpool; ²Nanyang Technological University; ³WMG, University of Warwick; ⁴The Chinese University of Hong Kong

Arxiv: http://arxiv.org/pdf/2412.04292v1

Methods	Year	Real		Fully synthetic		Tampered		Overall
Methods	Year	Acc	F1	Acc	F1	Acc	F1	Acc	F1
AntifakePrompt [5]	2024	64.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 24.1)}}$	78.6 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 10.5)}}$	93.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 3.7)}}$	96.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 1.1)}}$	30.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 60.1)}}$	47.2 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 33.2)}}$	63.1 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 29.1)}}$	69.3 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 23.4)}}$
CnnSpott [17]	2021	79.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 9.2)}}$	88.7 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 2.1)}}$	39.5 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 51.2)}}$	56.6 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 31.5)}}$	6.9 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 61.2)}}$	12.9 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 51.1)}}$	42.1 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 39.3)}}$	69.6 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 20.7)}}$
FreDect [18]	2020	83.7 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 37.7)}}$	91.1 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 43.5)}}$	16.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 44.1)}}$	28.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 37.2)}}$	11.9 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 25.2)}}$	21.3 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 31.7)}}$	37.4 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 33.6)}}$	23.4 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 47.2)}}$
Fusing [27]	2022	85.1 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 4.1)}}$	92.0 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 0.7)}}$	34.0 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 54.1)}}$	50.7 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 38.4)}}$	2.7 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 24.3)}}$	5.3 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 26.1)}}$	40.1 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 33.1)}}$	29.1 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 40.3)}}$
Gram-Net [39]	2020	70.1 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 19.1)}}$	82.4 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 9.3)}}$	93.5 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 4.4)}}$	96.6 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 2.0)}}$	0.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 89.1)}}$	1.6 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 85.3)}}$	55.0 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 37.1)}}$	58.0 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 35.1)}}$
UnivFD [48]	2023	68.0 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 0.3)}}$	67.4 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 1.1)}}$	62.1 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 24.3)}}$	87.5 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 10.5)}}$	64.0 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 28.5)}}$	85.3 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 4.7)}}$	64.0 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 21.7)}}$	85.3 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 4.5)}}$ 0
LGrad [61]	2023	64.8 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 2.8)}}$	78.6 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 2.5)}}$	83.5 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 25.5)}}$	91.0 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 23.7)}}$	6.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow\textbf{92.3})}}$	12.7 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow\textbf{86.1})}}$	51.8 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 20.2)}}$	55.5 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 23.9)}}$
LNP [2]	2023	71.2 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 56.8)}}$	83.2 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 60.2)}}$	91.8 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 55.6)}}$	95.7 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 60.1)}}$	2.9 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 90.4)}}$	5.7 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 88.9)}}$	55.2 ${{\color[rgb]{0,1,0}\scriptstyle(\downarrow 7.6)}}$	58.2 ${{\color[rgb]{1,0,0}\scriptstyle(\uparrow 4.1)}}$ 0
SIDA-7B	2024	96.6	96.1	99.3	99.7	95.6	96.2	97.3	97.5
SIDA-13B	2024	96.7	97.3	98.7	99.3	97.8	96.7	97.6	97.8