notesum.ai

Published at November 18

Dissecting Misalignment of Multimodal Large Language Models via Influence Function

cs.AI

cs.CV

Released Date: November 18, 2024

Authors: Lijie Hu¹, Chenyang Ren², Huanyi Xie³, Khouloud Saadi¹, Shu Yang¹, Jingfeng Zhang⁴, Di Wang¹

Aff.: ¹King Abdullah University of Science and Technology; ²Shanghai Jiao Tong University; ³KTH Royal Institute of Technology; ⁴The University of Auckland

Arxiv: http://arxiv.org/abs/2411.11667v1

Sample	Method	FGVCAircraft		Food101		Flowers102
Sample	Method	Accuracy(%)	RT (second)	Accuracy(%)	RT (second)	Accuracy(%)	RT (second)
Random	Retrain	23.07 $\pm$ 0.29	19.57	84.93 $\pm$ 0.17	14.59	68.16 $\pm$ 0.22	16.59
Random	ECIF	22.77 $\pm$ 0.09	7.60	84.87 $\pm$ 0.24	7.28	68.53 $\pm$ 0.12	7.29
Valuable	Retrain	22.93 $\pm$ 0.33	15.56	84.80 $\pm$ 0.16	15.88	68.23 $\pm$ 0.33	16.43
Valuable	ECIF	22.73 $\pm$ 0.09	5.95	84.86 $\pm$ 0.05	6.27	68.26 $\pm$ 0.12	6.52
Harmful	Retrain	23.50 $\pm$ 0.11	22.40	84.83 $\pm$ 0.05	14.59	68.00 $\pm$ 0.16	16.09
Harmful	ECIF	23.02 $\pm$ 0.07	6.26	84.90 $\pm$ 0.01	6.22	68.30 $\pm$ 0.01	6.27