notesum.ai
Published at November 8Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?
cs.CL
cs.AI
Released Date: November 8, 2024
Authors: Veronica Chatrath1, Marcelo Lotif1, Shaina Raza1
Aff.: 1Vector Institute for Artificial Intelligence
| Method | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|
| Llama-3-8B-Instruct (zero-shot) | 80.4% | 82.3% | 80.4% | 80.7% |
| Llama-3-8B-Instruct (5-shot) | 89.3% | 89.3% | 89.3% | 89.2% |
| Llama-3.1-8B-Instruct (zero-shot) | 74.5% | 81.0% | 74.5% | 74.7% |
| Llama-3.1-8B-Instruct (5-shot) | 82.1% | 86.4% | 82.1% | 82.3% |
| Mistral-7B-Instruct-v0.3 (zero-shot) | 80.0% | 79.7% | 80.0% | 79.7% |
| Mistral-7B-Instruct-v0.3 (5-shot) | 82.5% | 83.1% | 82.5% | 81.8% |
| Gemma-2-9b-Instruct (zero-shot) | 77.6% | 82.0% | 77.6% | 77.9% |
| Gemma-2-9b-Instruct (5-shot) | 83.3% | 85.4% | 83.3% | 83.6% |
| Phi-3-med-128k-Instruct (zero-shot) | 80.7% | 80.8% | 80.7% | 80.7% |
| Phi-3-med-128k-Instruct (5-shot) | 86.9% | 87.1% | 86.9% | 86.7% |