notesum.ai
Published at December 10Automatic Database Configuration Debugging using Retrieval-Augmented Language Models
cs.DB
Released Date: December 10, 2024
Authors: Sibei Chen1, Ju Fan, Bin Wu, Nan Tang, Chao Deng, Pengyi Wang, Ye Li, Jian Tan, Feifei Li, Jingren Zhou, Xiaoyong Du
Aff.: 1Renmin University of China

| Methods | Settings | MySQL SO | MySQL Forum | PG SO | MySQL Run | ||||||
| Recall | Precision | F1-score | Recall | Precision | F1-score | Recall | Precision | F1-score | SuccessRate | ||
| PLM | PLM | 0.529 | 0.004 | 0.008 | 0.547 | 0.008 | 0.015 | 0.558 | 0.003 | 0.006 | - |
| PLM+DA | 0.518 | 0.004 | 0.008 | 0.587 | 0.009 | 0.017 | 0.485 | 0.005 | 0.01 | - | |
| LLMs | GPT-3.5 | 0.329 | 0.205 | 0.221 | 0.207 | 0.145 | 0.153 | 0.32 | 0.183 | 0.201 | 0.37 |
| GPT-4 | 0.291 | 0.201 | 0.222 | 0.143 | 0.052 | 0.06 | 0.272 | 0.14 | 0.161 | 0.43 | |
| Llama8B | 0.07 | 0.025 | 0.035 | 0.078 | 0.021 | 0.033 | 0.056 | 0.023 | 0.032 | 0.25 | |
| Llama70B | 0.054 | 0.027 | 0.034 | 0.051 | 0.026 | 0.034 | 0.091 | 0.031 | 0.045 | 0.26 | |
| Qwen7B | 0.024 | 0.024 | 0.019 | 0.008 | 0.006 | 0.007 | 0.044 | 0.015 | 0.021 | 0.11 | |
| Qwen72B | 0.208 | 0.125 | 0.140 | .0.171 | 0.112 | 0.116 | 0.114 | 0.05 | 0.065 | 0.31 | |
| LLMs (all knobs) | GPT-3.5 | 0.448 | 0.293 | 0.308 | 0.411 | 0.247 | 0.246 | 0.465 | 0.268 | 0.299 | 0.52 |
| GPT-4 | 0.507 | 0.300 | 0.332 | 0.371 | 0.227 | 0.235 | 0.504 | 0.38 | 0.390 | 0.49 | |
| Llama8B | 0.116 | 0.048 | 0.057 | 0.11 | 0.054 | 0.058 | 0.263 | 0.094 | 0.121 | 0.31 | |
| Llama70B | 0.536 | 0.217 | 0.268 | 0.368 | 0.166 | 0.201 | 0.488 | 0.25 | 0.277 | 0.57 | |
| Qwen7B | 0.103 | 0.086 | 0.087 | 0.136 | 0.164 | 0.137 | 0.232 | 0.247 | 0.222 | 0.10 | |
| Qwen72B | 0.451 | 0.3 | 0.321 | 0.384 | 0.248 | 0.26 | 0.521 | 0.293 | 0.305 | 0.54 | |
| LLMs (PE) | CoT | 0.308 | 0.170 | 0.196 | 0.161 | 0.115 | 0.121 | 0.226 | 0.144 | 0.158 | 0.53 |
| Task Dec. | 0.143 | 0.136 | 0.127 | 0.111 | 0.160 | 0.100 | 0.207 | 0.169 | 0.179 | 0.5 | |
| LLM (SFT) | GPT-3.5 | 0.345 | 0.391 | 0.347 | 0.333 | 0.6 | 0.390 | 0.343 | 0.383 | 0.347 | 0.46 |
| Andromeda | GPT-3.5 | 0.512 | 0.318 | 0.348 | 0.541 | 0.383 | 0.382 | 0.461 | 0.305 | 0.321 | 0.79 |
| GPT-4 | 0.557 | 0.426 | 0.441 | 0.528 | 0.495 | 0.449 | 0.47 | 0.402 | 0.398 | 0.76 | |
| Llama8B | 0.32 | 0.183 | 0.201 | 0.441 | 0.244 | 0.256 | 0.39 | 0.241 | 0.270 | 0.44 | |
| Llama70B | 0.507 | 0.315 | 0.345 | 0.482 | 0.331 | 0.347 | 0.452 | 0.31 | 0.335 | 0.60 | |
| Qwen7B | 0.294 | 0.245 | 0.242 | 0.253 | 0.192 | 0.195 | 0.275 | 0.225 | 0.222 | 0.46 | |
| Qwen72B | 0.529 | 0.435 | 0.437 | 0.488 | 0.45 | 0.412 | 0.449 | 0.415 | 0.410 | 0.60 | |