notesum.ai

Published at December 6

cs.CL

Released Date: December 6, 2024

Authors: James Beetham¹, Souradip Chakraborty², Mengdi Wang³, Furong Huang², Amrit Singh Bedi¹, Mubarak Shah¹

Aff.: ¹University of Central Florida; ²University of Maryland, College Park; ³Princeton University

TargetLLM	Attack	ASR@ $1/10/100$	Perplexity	TTA1 $/$ TTA100
Vicuna-7b	GCG (individual)	$99.10/-/-$	$92471.12$	16m $/$ 25h
	AutoDAN (individual)	$92.70/-/-$	$83.17$	15m $/$ 23h
	AdvPrompter	$26.92/84.61/99.04$	$12.09$	22h $/$ 22h
	LIAR (ours)	$12.55/53.08/97.12$	$\mathbf{2.14}$	45s $/$ 14m
Vicuna-13b	GCG (individual)	$95.40/-/-$	$94713.43$	16m $/$ 25h
	AutoDAN (individual)	$80.30/-/-$	$89.14$	15m $/$ 23h
	AdvPrompter	$19.50/67.50/-$	$15.91$	22h $/$ 22h
	LIAR (ours)	$0.94/31.35/79.81$	$\mathbf{2.12}$	45s $/$ 14m
Llama2-7b	GCG (individual)	$23.70/-/-$	$97381.10$	16m $/$ 25h
	AutoDAN (individual)	$20.90/-/-$	$429.12$	15m $/$ 23h
	AdvPrompter	$1.00/7.70/-$	$86.80$	22h $/$ 22h
	LIAR (ours)	$0.65/2.31/3.85$	$\mathbf{2.13}$	45s $/$ 14m
Mistral-7b	GCG (individual)	$100.0/-/-$	$81432.10$	16m $/$ 25h
	AutoDAN (individual)	$91.20/-/-$	$69.09$	15m $/$ 23h
	AdvPrompter	$54.30/96.10/-$	$41.60$	22h $/$ 22h
	LIAR (ours)	$34.25/73.94/96.15$	$\mathbf{2.12}$	45s $/$ 14m
Falcon-7b	GCG (individual)	$100.0/-/-$	$94371.10$	16m $/$ 25h
	AutoDAN (individual)	$100.0/-/-$	$16.46$	15m $/$ 23h
	AdvPrompter	$78.80/98.10/-$	10	22h $/$ 22h
	LIAR (ours)	$71.78/99.33/100.0$	$\mathbf{2.07}$	45s $/$ 14m
Pythia-7b	GCG (individual)	$100.0/-/-$	$107346.41$	16m $/$ 25h
	AutoDAN (individual)	$100.0/-/-$	$16.05$	15m $/$ 23h
	AdvPrompter	$80.30/100.0/-$	$7.16$	22h $/$ 22h
	LIAR (ours)	$75.96/99.81/100.0$	$\mathbf{2.17}$	45s $/$ 14m