notesum.ai

Published at November 21

GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs

cs.AI

cs.CR

cs.CV

Released Date: November 21, 2024

Authors: Advik Raj Basani¹, Xiao Zhang²

Aff.: ¹BITS Pilani, Goa Campus, India; ²CISPA Helmholtz Center for Information Security, Germany

Arxiv: http://arxiv.org/abs/2411.14133v1

Refer to caption

Attack	Evaluator	TargetLLM (ASR@1 / ASR@10)
Attack	Evaluator	Mistral-7b-v0.3	Falcon-7b	Llama-3.1-8b	Llama-3-8b	Llama-2-7b
GCG	Keyword Matching	$-/47$	$-/75$	$-/6$	$-/0$	$-/3$
	StrongREJECT	$-/22$	$-/17$	$-/7$	$-/8$	$-/17$
	GASPEval	$-/37$	$-/52$	$-/6$	$-/2$	$-/5$
AutoDAN	Keyword Matching	$-/100^{*}$	$-/69$	$-/2$	$-/100^{*}$	$-/1$
	StrongREJECT	$-/64$	$-/34$	$-/2$	$-/54$	$-/1$
	GASPEval	$-/69$	$-/42$	$-/1$	$-/62$	$-/0$
AdvPrompter	Keyword Matching	$38/52$	$73/95$	$0/11$	$1/7$	$1/6$
	StrongREJECT	$48/71$	$51/92$	$4/13$	$0/8$	$1/4$
	GASPEval	$55/77$	$52/93$	$4/17$	$0/5$	$1/7$
GASP	Keyword Matching	$31/58$	$72/94$	$6/20$	$0/1$	$0/0^{*}$
	StrongREJECT	$56/84$	$53/99$	$7/31$	$9/34$	$15/31$
	GASPEval	$64/82$	$86/100$	$11/68$	$6/71$	$9/64$