notesum.ai

Published at December 10

FlexLLM: Exploring LLM Customization for Moving Target Defense on Black-Box LLMs Against Jailbreak Attacks

cs.CR

cs.CL

Released Date: December 10, 2024

Authors: Bocheng Chen¹, Hanqing Guo², Qiben Yan¹

Aff.: ¹Michigan State University; ²University of Hawaii at Manoa

Arxiv: http://arxiv.org/pdf/2412.07672v1

Refer to caption

Attack	Different Defenses							MTD
	ICD	PPL	Retokenization	SafeDecoding	Self-Exam	Self-Reminder	nodefense
	Dolphin-llama2-7b
PAIR	0.50	0.56	0.68	0.54	0.46	0.62	0.56	0.32
AutoDAN	0.08	0.26	0.68	0.26	0.20	0.36	0.26	0.10
GCG	0.46	0.00	0.50	0.38	0.34	0.58	0.38	0.16
DeepInception	0.20	0.10	0.74	0.10	0.06	0.90	0.10	0.00
Average	0.31	0.23	0.65	0.32	0.27	0.62	0.33	0.15
	Vicuna-7b
PAIR	0.10	0.18	0.06	0.18	0.04	0.06	0.18	0.06
AutoDAN	0.02	0.08	0.00	0.08	0.00	0.10	0.08	0.04
GCG	0.00	0.00	0.00	0.08	0.02	0.00	0.08	0.00
DeepInception	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Average	0.03	0.07	0.02	0.09	0.02	0.04	0.09	0.03
	Llama2-7b-chat
PAIR	0.02	0.12	0.04	0.12	0.12	0.02	0.12	0.02
AutoDAN	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
GCG	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
DeepInception	0.00	0.00	0.04	0.00	0.00	0.00	0.00	0.00
Average	0.01	0.03	0.02	0.03	0.03	0.01	0.03	0.01
	Guanaco-7b
PAIR	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
AutoDAN	0.00	0.00	0.12	0.00	0.00	0.00	0.00	0.00
GCG	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
DeepInception	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Average	0.00	0.00	0.03	0.00	0.00	0.00	0.00	0.00
	Falcon-7b
PAIR	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
AutoDAN	0.00	0.00	0.16	0.00	0.00	0.00	0.00	0.00
GCG	0.00	0.00	0.02	0.00	0.00	0.00	0.00	0.00
DeepInception	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Average	0.00	0.00	0.05	0.00	0.00	0.00	0.00	0.00