notesum.ai
Published at November 22KBAda: Efficient Self Adaptation on Specific Knowledge Bases
cs.CL
cs.AI
Released Date: November 22, 2024
Authors: Zheni Zeng1, Yuxuan Chen1, Shi Yu1, Yukun Yan1, Zhenghao Liu2, Shuo Wang1, Xu Han1, Zhiyuan Liu1, Maosong Sun1
Aff.: 1Tsinghua University; 2Northeastern University

| Methods | LooGLE | ASQA | JEC-QA | ||||||||||
| F1 | BLEU | ROUGE | BERT | LLM | Match | BLEU | ROUGE | BERT | LLM | Single | Multi | Total | |
| GPT series | |||||||||||||
| GPT-3.5 | 35.42 | 13.11 | 32.67 | 80.99 | 78.08 | 26.79 | 3.19 | 25.37 | 86.61 | 51.66 | 14.49 | 17.92 | 16.32 |
| wo QE | 35.27 | 12.97 | 32.51 | 80.94 | 77.91 | 27.40 | 3.13 | 24.92 | 86.71 | 51.52 | 13.84 | 19.15 | 16.68 |
| GPT-4o | 40.20 | 14.93 | 36.07 | 81.70 | 82.93 | 32.18 | 2.87 | 27.69 | 87.14 | 67.76 | 21.95 | 26.42 | 24.33 |
| wo QE | 40.21 | 15.18 | 36.06 | 81.71 | 83.29 | 32.15 | 2.94 | 27.49 | 87.10 | 67.88 | 20.11 | 27.36 | 23.98 |
| MiniCPM-2.4B | |||||||||||||
| Original | 30.92 | 10.73 | 29.27 | 80.70 | 64.76 | 11.91 | 1.67 | 21.84 | 82.30 | 22.92 | 39.24 | 13,87 | 25.69 |
| wo QE | 30.31 | 10.50 | 28.62 | 80.37 | 64.72 | 12.37 | 1.72 | 22.36 | 82.90 | 22.42 | 38.38 | 14.06 | 25.39 |
| Ours | 54.09 | 18.32 | 49.75 | 86.48 | 75.19 | 15.68 | 2.67 | 24.59 | 85.41 | 24.81 | 49.95 | 9.94 | 28.91 |
| (+23.17) | (+7.59) | (+20.48) | (+5.78) | (+10.43) | (+3.77) | (+1.00) | (+2.75) | (+3.11) | (+1.89) | (+10.71) | (-3.93) | (+3.22) | |
| wo QE | 53.76 | 18.55 | 49.61 | 86.23 | 73.19 | 16.12 | 2.75 | 24.65 | 85.48 | 25.69 | 49.41 | 10.92 | 29.16 |
| LLaMA3.1-8B-Instruction | |||||||||||||
| Original | 40.46 | 15.91 | 36.31 | 81.57 | 77.15 | 20.21 | 2.94 | 23.24 | 84.93 | 37.28 | 22.70 | 24.66 | 23.73 |
| wo QE | 39.94 | 15.78 | 36.14 | 81.50 | 77.08 | 20.03 | 2.94 | 23.09 | 85.14 | 35.64 | 22.92 | 24.07 | 23.53 |
| Ours | 62.07 | 21.73 | 57.34 | 88.63 | 80.16 | 25.23 | 3.43 | 23.59 | 86.29 | 42.44 | 34.59 | 14.13 | 23.83 |
| (+21.61) | (+5.82) | (+21.03) | (+6.06) | (+2.85) | (+5.02) | (+0.49) | (+1.35) | (+1.36) | (+5.16) | (+11.89) | (-10.53) | (+0.10) | |
| wo QE | 61.79 | 21.60 | 57.09 | 88.55 | 79.96 | 25.56 | 3.23 | 23.27 | 86.89 | 41.31 | 34.16 | 14.42 | 23.78 |