notesum.ai
Published at November 4Can Language Models Enable In-Context Database?
cs.DB
cs.AI
Released Date: November 4, 2024
Authors: Yu Pan1, Hongfeng Yu, Tianjiao Zhao2, Jianxin Sun
Aff.: 1University of Nebraska-Lincoln; 2East Carolina University
| Prompting | Encoding | Update | Delete | Insert | Select |
| GPT-4o | |||||
| ZERO-SHOT | SQL | 22.73% | 23.06% | 21.33% | 42.01% |
| NL | 49.83% | 28.54% | 23.45% | 20.64% | |
| ZERO-COT | SQL | 19.35% | 16.41% | 18.38% | 35.84% |
| NL | 30.77% | 24.33% | 20.9% | 17.77% | |
| FEW-SHOT | SQL | 43.14% | 55.77% | 86.89% | 65.39% |
| NL | 87.96% | 45.17% | 82.88% | 52.23% | |
| LLama3.1-8B | |||||
| ZERO-SHOT | SQL | 4.31% | 8.65% | 6.55% | 3.28% |
| NL | 5.26% | 7.99% | 7.70% | 1.66% | |
| ZERO-COT | SQL | 2.35% | 2.81% | 2.19% | 3.17% |
| NL | 2.49% | 2.77% | 2.84% | 1.58% | |
| FEW-SHOT | SQL | 43.48% | 55.97% | 55.57% | 29.96% |
| NL | 39.46% | 47.29% | 50.36% | 28.85% | |
| Mistral | |||||
| ZERO-SHOT | SQL | 1.78% | 1.97% | 1.94% | 1.40% |
| NL | 1.94% | 2.25% | 2.19% | 0.72% | |
| ZERO-COT | SQL | 0.86% | 1.13% | 0.9% | 1.28% |
| NL | 0.91% | 1.65% | 1.33% | 0.54% | |
| FEW-SHOT | SQL | 11.0% | 16.29% | 15.66% | 24.52% |
| NL | 14.67% | 39.75% | 24.51% | 24.13% | |
| Gemma2-9B | |||||
| ZERO-SHOT | SQL | 6.96% | 14.28% | 13.02% | 4.82% |
| NL | 8.49% | 20.74% | 15.4% | 2.34% | |
| ZERO-COT | SQL | 2.69% | 2.4% | 4.01% | 4.90% |
| NL | 2.59% | 3.74% | 3.35% | 2.39% | |
| FEW-SHOT | SQL | 86.32% | 52.12% | 66.72% | 29.65% |
| NL | 46.17% | 49.12% | 77.15% | 29.79% | |
| LLama3.2-3B | |||||
| ZERO-SHOT | SQL | 6.88% | 17.25% | 10.20% | 4.11% |
| NL | 7.32% | 19.36% | 12.90% | 2.15% | |
| ZERO-COT | SQL | 2.68% | 2.12% | 3.12% | 3.53% |
| NL | 3.02% | 3.07% | 2.89% | 1.79% | |
| FEW-SHOT | SQL | 52.52% | 54.13% | 61.21% | 28.25% |
| NL | 56.91% | 56.06% | 63.10% | 17.20% | |
| Finetuned LLama3.1-8B | |||||
| FEW-SHOT | SQL | 75.0% | 55.0% | 88.20% | 48.82% |
| NL | 49.83% | 56.21% | 90.24% | 46.41% | |