notesum.ai
Published at November 26On Limitations of LLM as Annotator for Low Resource Languages
cs.CL
cs.LG
Released Date: November 26, 2024
Authors: Suramya Jadhav1, Abhay Shanbhag1, Amogh Thakurdesai1, Ridhima Sinare1, Raviraj Joshi2
Aff.: 1Pune Institute of Computer Technology, Pune; 2Indian Institute of Technology Madras, Chennai
| Dataset | Tech | Llama 3.1 8B | Gemma 2 2B | Gemma 2 9B | Gemini 1.0 Pro | GPT-4o | BERT |
|---|---|---|---|---|---|---|---|
| MahaSent | ZS | 0.76 | 0.71 | 0.69 | 0.78 | 0.79 | 0.80 |
| FS | 0.79 | 0.76 | 0.78 | 0.76 | 0.82 | ||
| MahaHate-2C | ZS | 0.64 | 0.71 | 0.78 | 0.74 | 0.80 | 0.91 |
| FS | 0.78 | 0.72 | 0.82 | 0.72 | 0.82 | ||
| MahaHate-4C | ZS | 0.40 | 0.39 | 0.43 | 0.43 | 0.58 | 0.73 |
| FS | 0.48 | 0.41 | 0.46 | 0.45 | 0.60 | ||
| MahaNews-SHC | ZS | 0.60 | 0.54 | 0.68 | 0.68 | 0.78 | 0.85 |
| FS | 0.66 | 0.54 | 0.68 | 0.70 | 0.78 | ||
| MahaNews-LPC | ZS | 0.66 | 0.55 | 0.71 | 0.72 | 0.77 | 0.89 |
| FS | 0.67 | 0.50 | 0.72 | 0.74 | 0.75 | ||
| MahaNews-LDC | ZS | 0.69 | 0.62 | 0.78 | 0.74 | 0.81 | 0.96 |
| FS | 0.69 | 0.62 | 0.80 | 0.75 | 0.81 |