notesum.ai

Published at October 30

ProTransformer: Robustify Transformers via Plug-and-Play Paradigm

cs.LG

cs.CL

cs.CR

Released Date: October 30, 2024

Authors: Zhichao Hou¹, Weizhi Gao¹, Yuchen Shen², Feiyi Wang³, Xiaorui Liu¹

Aff.: ¹North Carolina State University; ²Carnegie Mellon University; ³Oak Ridge National Laboratory

Arxiv: http://arxiv.org/abs/2410.23182v1

Refer to caption

		Textfooler		TextBugger		DeepWordBug		PWWS
Model	Clean% $\uparrow$	Aua% $\uparrow$	ASR% $\downarrow$	AUA% $\uparrow$	ASR% $\downarrow$	AUA% $\uparrow$	ASR% $\downarrow$	AUA% $\uparrow$	ASR% $\downarrow$
ALBERT	93.0	20.6	77.9	26.1	71.9	38.9	58.2	35.9	61.4
Pro-ALBERT (MCP) (Ours)	93.8	48.9	47.3	41.8	55.3	59.5	35.9	63.1	32.0
DistilBERT	93.5	13.2	85.9	33.6	63.4	30.0	67.9	36.5	61.0
Pro-DistilBERT (MCP) (Ours)	93.9	29.3	68.5	48.7	47.9	34.3	63.1	50.5	45.6
RoBERTa	93.4	13.0	86.1	32.5	64.5	41.2	55.9	34.0	63.6
Pro-RoBERTa (MCP) (Ours)	93.7	24.4	73.7	34.3	62.8	45.5	51.5	39.4	57.5
BERT	94.2	19.7	78.9	31.7	67.5	37.5	59.8	43.1	53.8
+ FreeLB	94.2	38.0	59.5	42.8	55.5	56.1	40.9	57.0	39.9
+ PGD	94.1	36.8	61.7	40.5	57.1	47.6	49.7	48.7	48.6
+ MixADA	94.3	35.6	62.4	35.4	62.9	38.2	50.5	46.8	50.4
+ TA-VAT	94.4	36.2	61.8	39.2	58.2	49.5	48.1	47.0	50.7
+ AT	94.1	42.1	54.8	56.1	39.4	42.4	54.1	62.6	32.5
Pro-BERT ( $\ell_{1}$ ) (Ours)	94.2	23.8	74.5	43.8	53.0	48.7	47.8	46.5	50.1
Pro-BERT (Huber) (Ours)	94.2	24.2	74.0	43.7	52.9	46.0	50.5	48.4	47.9
Pro-BERT (MCP) (Ours)	93.2	39.2	57.7	48.3	48.5	51.8	43.8	56.2	39.2
Pro-BERT (MCP) + AT (Ours)	94.0	56.8	38.9	60.7	35.1	61.0	34.1	68.8	25.7