notesum.ai
Published at December 10MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMs
cs.CR
cs.LG
Released Date: December 10, 2024
Authors: Zhenpeng Wu1, Jian Lou2, Zibin Zheng2, Chuan Chen1
Aff.: 1Sun Yat-Sen University, Guangzhou, China; 2Sun Yat-Sen University, Zhuhai, China

| Method | 0.48s (5 trials) | 0.96s (10 trials) | 1.92s (20 trials) | 2.88s (30 trials) | ||||
|---|---|---|---|---|---|---|---|---|
| Avg LCSS | Hit Rate | Avg LCSS | Hit Rate | Avg LCSS | Hit Rate | Avg LCSS | Hit Rate | |
| Train Set | ||||||||
| Manual (Carlini et al., 2023) | 48.74 | 12.50 | 48.74 | 12.50 | 48.74 | 12.50 | 48.74 | 12.50 |
| MiniPrompt (Schwarzschild et al., 2024) | 46.19 | 0.00 | 49.06 | 0.00 | 54.36 | 0.00 | 61.80 | 0.00 |
| MemHunter (ours) | 77.62 | 35.16 | 81.50 | 37.73 | 84.52 | 44.32 | 87.63 | 47.62 |
| Test Set | ||||||||
| Manual (Carlini et al., 2023) | 49.81 | 10.00 | 49.81 | 10.00 | 49.81 | 10.00 | 49.81 | 10.00 |
| MiniPrompt (Schwarzschild et al., 2024) | 49.00 | 0.00 | 50.93 | 0.00 | 51.48 | 0.00 | 61.90 | 0.00 |
| MemHunter (ours) | 77.85 | 30.38 | 79.31 | 32.91 | 85.91 | 37.97 | 84.81 | 43.04 |