notesum.ai
Published at November 27Neutralizing Backdoors through Information Conflicts for Large Language Models
cs.CL
Released Date: November 27, 2024
Authors: Chen Chen1, Yuchen Sun2, Xueluan Gong1, Jiaxin Gao2, Kwok-Yan Lam1
Aff.: 1Nanyang Technological University, Singapore; 2Wuhan University, China
| Model | Attack | Metrics | Backdoored | Editing | Wanda | Fine-tuning | Fine-pruning | Speculative | NAD | BEEAR | Ours |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT2-XL | CBA | ASR | 100.0% | 98.75% | 99.37% | 100.0% | 37.51% | 98.00% | 29.67% | 28.56% | 1.26% |
| CDA | 91.57% | 89.53% | 88.07% | 93.88% | 91.31% | 90.32% | 89.74% | 91.66% | 88.89% | ||
| BadEdit | ASR | 98.36% | 90.12% | 91.66% | 1.47% | 26.68% | 98.36% | 7.77% | 2.25% | 0.00% | |
| CDA | 87.27% | 90.19% | 77.90% | 91.97% | 87.97% | 88.82% | 97.18% | 93.91% | 86.30% | ||
| Rome | ASR | 99.54% | 27.60% | 99.32% | 69.44% | 36.06% | 98.90% | 9.59% | 17.75% | 0.51% | |
| CDA | 57.91% | 50.17% | 60.43% | 73.85% | 56.97% | 56.87% | 56.57% | 85.92% | 62.98% | ||
| MEMIT | ASR | 100.0% | 63.98% | 97.41% | 13.85% | 63.55% | 100.0% | 11.24% | 19.60% | 0.13% | |
| CDA | 57.79% | 59.96% | 60.43% | 81.31% | 59.16% | 58.33% | 53.11% | 92.07% | 70.28% | ||
| LWP | ASR | 56.72% | 53.11% | 55.79% | 19.80% | 10.43% | 53.28% | 6.45% | 42.99% | 0.57% | |
| CDA | 90.49% | 91.03% | 86.37% | 94.36% | 93.76% | 91.62% | 86.7% | 88.50% | 90.70% | ||
| GPT-J | CBA | ASR | 100.0% | 97.19% | 80.64% | 78.94% | 60.63% | 98.82% | 31.05% | 13.56% | 1.07% |
| CDA | 90.43% | 90.7% | 87.72% | 93.18% | 91.15% | 91.49% | 87.22% | 92.57% | 91.33% | ||
| BadEdit | ASR | 98.85% | 18.37% | 86.52% | 1.88% | 15.47% | 97.39% | 9.85% | 1.44% | 2.26% | |
| CDA | 71.67% | 82.90% | 70.35% | 91.10% | 73.82% | 78.59% | 69.08% | 80.90% | 74.19% | ||
| Rome | ASR | 100% | 23.9% | 89.37% | 0.00% | 34.34% | 99.67% | 2.18% | 4.18% | 2.79% | |
| CDA | 72.85% | 70.14% | 79.61% | 90.08% | 67.75% | 74.14% | 69.18% | 84.11% | 73.27% | ||
| MEMIT | ASR | 99.08% | 49.51% | 89.17% | 8.56% | 65.09% | 97.22% | 13.56% | 6.49% | 4.11% | |
| CDA | 71.55% | 76.20% | 83.26% | 96.94% | 74.10% | 71.86% | 72.83% | 75.97% | 74.31% | ||
| LWP | ASR | 65.15% | 55.68% | 41.60% | 16.25% | 30.57% | 64.78% | 3.74% | 1.35% | 3.90% | |
| CDA | 89.14% | 79.08% | 77.11% | 90.92% | 88.82% | 89.02% | 90.46% | 91.39% | 90.33% | ||
| Llama | CBA | ASR | 74.00% | 73.67% | 57.98% | 94.76% | 29.61% | 72.10% | 8.09% | 53.81% | 0.78% |
| CDA | 92.79% | 90.93% | 0.0% | 92.88% | 77.02% | 93.53% | 90.93% | 93.08% | 92.21% | ||
| BadEdit | ASR | 100.0% | 27.51% | 18.64% | 1.00% | 42.87% | 99.30% | 12.72% | 9.86% | 0.34% | |
| CDA | 66.16% | 59.39% | 51.83% | 95.64% | 68.14% | 65.49% | 62.85% | 83.49% | 72.35% | ||
| Rome | ASR | 99.15% | 21.29% | 17.03% | 28.75% | 14.31% | 97.94% | 9.16% | 6.82% | 0.53% | |
| CDA | 67.13% | 68.47% | 72.31% | 94.92% | 80.44% | 67.36% | 58.14% | 83.49% | 72.21% | ||
| MEMIT | ASR | 99.06% | 31.87% | 13.82% | 19.06% | 9.67% | 98.89% | 6.26% | 5.07% | 0.00% | |
| CDA | 60.71% | 59.37% | 51.03% | 95.72% | 79.77% | 62.80% | 63.66% | 75.81% | 62.64% | ||
| LWP | ASR | 69.24% | 65.02% | 21.98% | 15.76% | 27.70% | 69.99% | 6.06% | 4.41% | 3.38% | |
| CDA | 89.74% | 90.72% | 84.35% | 94.92% | 78.34% | 90.06% | 88.06% | 88.53% | 91.53% | ||
| Llama-2 | CBA | ASR | 100.0% | 97.56% | 95.86% | 100.0% | 33.17% | 100.0% | 12.91% | 35.59% | 7.51% |
| CDA | 91.44% | 90.27% | 93.19% | 94.08% | 87.28% | 91.86% | 85.87% | 92.15% | 93.76% | ||
| BadEdit | ASR | 100.0% | 79.42% | 67.64% | 4.59% | 31.57% | 99.71% | 43.51% | 5.47% | 3.33% | |
| CDA | 71.75% | 76.08% | 73.19% | 88.69% | 70.03% | 72.18% | 70.32% | 88.42% | 83.99% | ||
| Rome | ASR | 100.0% | 60.52% | 57.40% | 5.17% | 39.02% | 100.0% | 47.77% | 4.91% | 3.67% | |
| CDA | 68.21% | 81.85% | 78.58% | 91.20% | 77.32% | 65.66% | 72.03% | 75.82% | 80.50% | ||
| MEMIT | ASR | 100.0% | 71.66% | 57.33% | 0.00% | 54.36% | 92.70% | 35.23% | 6.81% | 9.33% | |
| CDA | 70.39% | 83.06% | 79.74% | 90.11% | 74.61% | 77.49% | 83.68% | 81.53% | 84.47% | ||
| LWP | ASR | 73.81% | 56.19% | 43.88% | 3.45% | 49.87% | 70.90% | 35.11% | 2.14% | 1.73% | |
| CDA | 86.92% | 85.47% | 90.65% | 91.02% | 87.57% | 88.41% | 80.03% | 88.73% | 89.36% |